2023-04-08 02:53:32,021 DEBUG [main] hbase.HBaseTestingUtility(342): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b 2023-04-08 02:53:32,036 INFO [main] hbase.HBaseClassTestRule(94): Test class org.apache.hadoop.hbase.TestAcidGuaranteesWithEagerPolicy timeout: 13 mins 2023-04-08 02:53:32,054 INFO [Time-limited test] hbase.HBaseTestingUtility(1068): Starting up minicluster with option: StartMiniClusterOption{numMasters=1, masterClass=null, numRegionServers=1, rsPorts=, rsClass=null, numDataNodes=1, dataNodeHosts=null, numZkServers=1, createRootDir=false, createWALDir=false} 2023-04-08 02:53:32,054 INFO [Time-limited test] hbase.HBaseZKTestingUtility(82): Created new mini-cluster data directory: /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b/cluster_2b175f16-edf4-6ecc-e1c4-da3bf5a7233b, deleteOnExit=true 2023-04-08 02:53:32,055 INFO [Time-limited test] hbase.HBaseTestingUtility(1082): STARTING DFS 2023-04-08 02:53:32,055 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting test.cache.data to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b/test.cache.data in system properties and HBase conf 2023-04-08 02:53:32,056 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.tmp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b/hadoop.tmp.dir in system properties and HBase conf 2023-04-08 02:53:32,056 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting hadoop.log.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b/hadoop.log.dir in system properties and HBase conf 2023-04-08 02:53:32,056 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.local.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b/mapreduce.cluster.local.dir in system properties and HBase conf 2023-04-08 02:53:32,057 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting mapreduce.cluster.temp.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b/mapreduce.cluster.temp.dir in system properties and HBase conf 2023-04-08 02:53:32,057 INFO [Time-limited test] hbase.HBaseTestingUtility(759): read short circuit is OFF 2023-04-08 02:53:32,174 WARN [Time-limited test] util.NativeCodeLoader(62): Unable to load native-hadoop library for your platform... using builtin-java classes where applicable 2023-04-08 02:53:32,603 DEBUG [Time-limited test] fs.HFileSystem(308): The file system is not a DistributedFileSystem. Skipping on block location reordering 2023-04-08 02:53:32,608 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-labels.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b/yarn.node-labels.fs-store.root-dir in system properties and HBase conf 2023-04-08 02:53:32,609 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.node-attribute.fs-store.root-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b/yarn.node-attribute.fs-store.root-dir in system properties and HBase conf 2023-04-08 02:53:32,609 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.log-dirs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b/yarn.nodemanager.log-dirs in system properties and HBase conf 2023-04-08 02:53:32,610 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-08 02:53:32,610 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.active-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b/yarn.timeline-service.entity-group-fs-store.active-dir in system properties and HBase conf 2023-04-08 02:53:32,611 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.timeline-service.entity-group-fs-store.done-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b/yarn.timeline-service.entity-group-fs-store.done-dir in system properties and HBase conf 2023-04-08 02:53:32,611 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting yarn.nodemanager.remote-app-log-dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b/yarn.nodemanager.remote-app-log-dir in system properties and HBase conf 2023-04-08 02:53:32,612 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-08 02:53:32,612 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.datanode.shared.file.descriptor.paths to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b/dfs.datanode.shared.file.descriptor.paths in system properties and HBase conf 2023-04-08 02:53:32,612 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting nfs.dump.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b/nfs.dump.dir in system properties and HBase conf 2023-04-08 02:53:32,613 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting java.io.tmpdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b/java.io.tmpdir in system properties and HBase conf 2023-04-08 02:53:32,613 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.journalnode.edits.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b/dfs.journalnode.edits.dir in system properties and HBase conf 2023-04-08 02:53:32,614 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting dfs.provided.aliasmap.inmemory.leveldb.dir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b/dfs.provided.aliasmap.inmemory.leveldb.dir in system properties and HBase conf 2023-04-08 02:53:32,614 INFO [Time-limited test] hbase.HBaseTestingUtility(772): Setting fs.s3a.committer.staging.tmp.path to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b/fs.s3a.committer.staging.tmp.path in system properties and HBase conf Formatting using clusterid: testClusterID 2023-04-08 02:53:33,219 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-08 02:53:33,224 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-08 02:53:33,535 WARN [Time-limited test] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-namenode.properties,hadoop-metrics2.properties 2023-04-08 02:53:33,729 INFO [Time-limited test] log.Slf4jLog(67): Logging to org.slf4j.impl.Reload4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog 2023-04-08 02:53:33,750 WARN [Time-limited test] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-08 02:53:33,794 INFO [Time-limited test] log.Slf4jLog(67): jetty-6.1.26 2023-04-08 02:53:33,825 INFO [Time-limited test] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/hdfs to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b/java.io.tmpdir/Jetty_localhost_38825_hdfs____rxlm00/webapp 2023-04-08 02:53:33,965 INFO [Time-limited test] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:38825 2023-04-08 02:53:33,981 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.heartbeat.interval(3) assuming SECONDS 2023-04-08 02:53:33,982 WARN [Time-limited test] conf.Configuration(1701): No unit for dfs.namenode.safemode.extension(0) assuming MILLISECONDS 2023-04-08 02:53:34,566 WARN [Listener at localhost/42901] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-08 02:53:34,649 WARN [Listener at localhost/42901] conf.Configuration(1701): No unit for dfs.datanode.outliers.report.interval(1800000) assuming MILLISECONDS 2023-04-08 02:53:34,674 WARN [Listener at localhost/42901] http.HttpRequestLog(97): Jetty request log can only be enabled using Log4j 2023-04-08 02:53:34,682 INFO [Listener at localhost/42901] log.Slf4jLog(67): jetty-6.1.26 2023-04-08 02:53:34,689 INFO [Listener at localhost/42901] log.Slf4jLog(67): Extract jar:file:/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/local-repository/org/apache/hadoop/hadoop-hdfs/2.10.0/hadoop-hdfs-2.10.0-tests.jar!/webapps/datanode to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b/java.io.tmpdir/Jetty_localhost_40617_datanode____.lnn515/webapp 2023-04-08 02:53:34,808 INFO [Listener at localhost/42901] log.Slf4jLog(67): Started HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:40617 2023-04-08 02:53:35,227 WARN [Listener at localhost/39741] common.MetricsLoggerTask(153): Metrics logging will not be async since the logger is not log4j 2023-04-08 02:53:35,777 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x1bf4b03560062fa3: Processing first storage report for DS-ed82e519-2bab-4f8a-8c3b-791831bebafc from datanode 14d5a713-17f1-418e-9bd0-2301d2c6ae84 2023-04-08 02:53:35,778 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x1bf4b03560062fa3: from storage DS-ed82e519-2bab-4f8a-8c3b-791831bebafc node DatanodeRegistration(127.0.0.1:36451, datanodeUuid=14d5a713-17f1-418e-9bd0-2301d2c6ae84, infoPort=38491, infoSecurePort=0, ipcPort=39741, storageInfo=lv=-57;cid=testClusterID;nsid=1577484136;c=1680922413296), blocks: 0, hasStaleStorage: true, processing time: 1 msecs, invalidatedBlocks: 0 2023-04-08 02:53:35,779 INFO [Block report processor] blockmanagement.BlockManager(2202): BLOCK* processReport 0x1bf4b03560062fa3: Processing first storage report for DS-db20bb6d-bff1-4821-b01e-a97b0aec7fb7 from datanode 14d5a713-17f1-418e-9bd0-2301d2c6ae84 2023-04-08 02:53:35,779 INFO [Block report processor] blockmanagement.BlockManager(2228): BLOCK* processReport 0x1bf4b03560062fa3: from storage DS-db20bb6d-bff1-4821-b01e-a97b0aec7fb7 node DatanodeRegistration(127.0.0.1:36451, datanodeUuid=14d5a713-17f1-418e-9bd0-2301d2c6ae84, infoPort=38491, infoSecurePort=0, ipcPort=39741, storageInfo=lv=-57;cid=testClusterID;nsid=1577484136;c=1680922413296), blocks: 0, hasStaleStorage: false, processing time: 0 msecs, invalidatedBlocks: 0 2023-04-08 02:53:35,785 DEBUG [Listener at localhost/39741] hbase.HBaseTestingUtility(649): Setting hbase.rootdir to /home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b 2023-04-08 02:53:35,864 INFO [Listener at localhost/39741] zookeeper.MiniZooKeeperCluster(258): Started connectionTimeout=30000, dir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b/cluster_2b175f16-edf4-6ecc-e1c4-da3bf5a7233b/zookeeper_0, clientPort=64826, secureClientPort=-1, dataDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b/cluster_2b175f16-edf4-6ecc-e1c4-da3bf5a7233b/zookeeper_0/version-2, dataDirSize=424 dataLogDir=/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b/cluster_2b175f16-edf4-6ecc-e1c4-da3bf5a7233b/zookeeper_0/version-2, dataLogSize=424 tickTime=2000, maxClientCnxns=300, minSessionTimeout=4000, maxSessionTimeout=40000, serverId=0 2023-04-08 02:53:35,879 INFO [Listener at localhost/39741] zookeeper.MiniZooKeeperCluster(283): Started MiniZooKeeperCluster and ran 'stat' on client port=64826 2023-04-08 02:53:35,886 INFO [Listener at localhost/39741] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-08 02:53:35,888 INFO [Listener at localhost/39741] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-08 02:53:36,560 INFO [Listener at localhost/39741] util.FSUtils(471): Created version file at hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01 with version=8 2023-04-08 02:53:36,560 INFO [Listener at localhost/39741] hbase.HBaseTestingUtility(1406): Setting hbase.fs.tmp.dir to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/hbase-staging 2023-04-08 02:53:36,905 INFO [Listener at localhost/39741] metrics.MetricRegistriesLoader(60): Loaded MetricRegistries class org.apache.hadoop.hbase.metrics.impl.MetricRegistriesImpl 2023-04-08 02:53:37,400 INFO [Listener at localhost/39741] client.ConnectionUtils(127): master/jenkins-hbase4:0 server-side Connection retries=45 2023-04-08 02:53:37,433 INFO [Listener at localhost/39741] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-08 02:53:37,434 INFO [Listener at localhost/39741] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-08 02:53:37,434 INFO [Listener at localhost/39741] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-08 02:53:37,434 INFO [Listener at localhost/39741] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-08 02:53:37,435 INFO [Listener at localhost/39741] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-08 02:53:37,593 INFO [Listener at localhost/39741] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.MasterService, hbase.pb.RegionServerStatusService, hbase.pb.LockService, hbase.pb.HbckService, hbase.pb.ClientMetaService, hbase.pb.ClientService, hbase.pb.AdminService 2023-04-08 02:53:37,669 DEBUG [Listener at localhost/39741] util.ClassSize(228): Using Unsafe to estimate memory layout 2023-04-08 02:53:37,769 INFO [Listener at localhost/39741] ipc.NettyRpcServer(120): Bind to /172.31.14.131:46135 2023-04-08 02:53:37,780 INFO [Listener at localhost/39741] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-08 02:53:37,782 INFO [Listener at localhost/39741] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-08 02:53:37,808 INFO [Listener at localhost/39741] zookeeper.RecoverableZooKeeper(93): Process identifier=master:46135 connecting to ZooKeeper ensemble=127.0.0.1:64826 2023-04-08 02:53:37,852 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:461350x0, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-08 02:53:37,855 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): master:46135-0x10161b07ca90000 connected 2023-04-08 02:53:37,883 DEBUG [Listener at localhost/39741] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-08 02:53:37,884 DEBUG [Listener at localhost/39741] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-08 02:53:37,888 DEBUG [Listener at localhost/39741] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-08 02:53:37,896 DEBUG [Listener at localhost/39741] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=46135 2023-04-08 02:53:37,897 DEBUG [Listener at localhost/39741] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=46135 2023-04-08 02:53:37,897 DEBUG [Listener at localhost/39741] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=46135 2023-04-08 02:53:37,898 DEBUG [Listener at localhost/39741] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=46135 2023-04-08 02:53:37,898 DEBUG [Listener at localhost/39741] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=46135 2023-04-08 02:53:37,904 INFO [Listener at localhost/39741] master.HMaster(444): hbase.rootdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01, hbase.cluster.distributed=false 2023-04-08 02:53:37,984 INFO [Listener at localhost/39741] client.ConnectionUtils(127): regionserver/jenkins-hbase4:0 server-side Connection retries=45 2023-04-08 02:53:37,984 INFO [Listener at localhost/39741] ipc.RpcExecutor(189): Instantiated default.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-08 02:53:37,984 INFO [Listener at localhost/39741] ipc.RpcExecutor(189): Instantiated priority.RWQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=2, maxQueueLength=30, handlerCount=3 2023-04-08 02:53:37,984 INFO [Listener at localhost/39741] ipc.RWQueueRpcExecutor(107): priority.RWQ.Fifo writeQueues=1 writeHandlers=1 readQueues=1 readHandlers=2 scanQueues=0 scanHandlers=0 2023-04-08 02:53:37,985 INFO [Listener at localhost/39741] ipc.RpcExecutor(189): Instantiated replication.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=3 2023-04-08 02:53:37,985 INFO [Listener at localhost/39741] ipc.RpcExecutor(189): Instantiated metaPriority.FPBQ.Fifo with queueClass=class java.util.concurrent.LinkedBlockingQueue; numCallQueues=1, maxQueueLength=30, handlerCount=1 2023-04-08 02:53:37,989 INFO [Listener at localhost/39741] ipc.RpcServerFactory(64): Creating org.apache.hadoop.hbase.ipc.NettyRpcServer hosting hbase.pb.ClientService, hbase.pb.AdminService 2023-04-08 02:53:37,992 INFO [Listener at localhost/39741] ipc.NettyRpcServer(120): Bind to /172.31.14.131:35089 2023-04-08 02:53:37,995 INFO [Listener at localhost/39741] hfile.BlockCacheFactory(142): Allocating BlockCache size=782.40 MB, blockSize=64 KB 2023-04-08 02:53:38,001 DEBUG [Listener at localhost/39741] mob.MobFileCache(120): MobFileCache enabled with cacheSize=1000, evictPeriods=3600sec, evictRemainRatio=0.5 2023-04-08 02:53:38,001 INFO [Listener at localhost/39741] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-08 02:53:38,004 INFO [Listener at localhost/39741] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-08 02:53:38,006 INFO [Listener at localhost/39741] zookeeper.RecoverableZooKeeper(93): Process identifier=regionserver:35089 connecting to ZooKeeper ensemble=127.0.0.1:64826 2023-04-08 02:53:38,009 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:350890x0, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=None, state=SyncConnected, path=null 2023-04-08 02:53:38,010 DEBUG [zk-event-processor-pool-0] zookeeper.ZKWatcher(623): regionserver:35089-0x10161b07ca90001 connected 2023-04-08 02:53:38,010 DEBUG [Listener at localhost/39741] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-08 02:53:38,012 DEBUG [Listener at localhost/39741] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-08 02:53:38,013 DEBUG [Listener at localhost/39741] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/acl 2023-04-08 02:53:38,013 DEBUG [Listener at localhost/39741] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=default.FPBQ.Fifo, numCallQueues=1, port=35089 2023-04-08 02:53:38,015 DEBUG [Listener at localhost/39741] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=priority.RWQ.Fifo.write, numCallQueues=1, port=35089 2023-04-08 02:53:38,015 DEBUG [Listener at localhost/39741] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=priority.RWQ.Fifo.read, numCallQueues=1, port=35089 2023-04-08 02:53:38,017 DEBUG [Listener at localhost/39741] ipc.RpcExecutor(311): Started handlerCount=3 with threadPrefix=replication.FPBQ.Fifo, numCallQueues=1, port=35089 2023-04-08 02:53:38,017 DEBUG [Listener at localhost/39741] ipc.RpcExecutor(311): Started handlerCount=1 with threadPrefix=metaPriority.FPBQ.Fifo, numCallQueues=1, port=35089 2023-04-08 02:53:38,020 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2168): Adding backup master ZNode /hbase/backup-masters/jenkins-hbase4.apache.org,46135,1680922416713 2023-04-08 02:53:38,029 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-08 02:53:38,031 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/backup-masters/jenkins-hbase4.apache.org,46135,1680922416713 2023-04-08 02:53:38,051 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-08 02:53:38,051 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/master 2023-04-08 02:53:38,051 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-08 02:53:38,052 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-08 02:53:38,053 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(227): Deleting ZNode for /hbase/backup-masters/jenkins-hbase4.apache.org,46135,1680922416713 from backup master directory 2023-04-08 02:53:38,053 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/master 2023-04-08 02:53:38,057 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/backup-masters/jenkins-hbase4.apache.org,46135,1680922416713 2023-04-08 02:53:38,057 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/backup-masters 2023-04-08 02:53:38,058 WARN [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-08 02:53:38,058 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ActiveMasterManager(237): Registered as active master=jenkins-hbase4.apache.org,46135,1680922416713 2023-04-08 02:53:38,062 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating data MemStoreChunkPool with chunk size 2 MB, max count 352, initial count 0 2023-04-08 02:53:38,063 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.ChunkCreator(497): Allocating index MemStoreChunkPool with chunk size 204.80 KB, max count 391, initial count 0 2023-04-08 02:53:38,154 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] util.FSUtils(620): Created cluster ID file at hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/hbase.id with ID: 620627a2-e57a-49c8-bcda-753bb9c42bc6 2023-04-08 02:53:38,202 INFO [master/jenkins-hbase4:0:becomeActiveMaster] fs.HFileSystem(337): Added intercepting call to namenode#getBlockLocations so can do block reordering using class org.apache.hadoop.hbase.fs.HFileSystem$ReorderWALBlocks 2023-04-08 02:53:38,217 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-08 02:53:38,259 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ReadOnlyZKClient(139): Connect 0x6ebbc202 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:53:38,294 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@226307fc, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:53:38,319 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(309): Create or load local region for table 'master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-08 02:53:38,321 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(132): Injected flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000 2023-04-08 02:53:38,338 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(264): ClientProtocol::create wrong number of arguments, should be hadoop 3.2 or below 2023-04-08 02:53:38,338 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(270): ClientProtocol::create wrong number of arguments, should be hadoop 2.x 2023-04-08 02:53:38,340 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputHelper(279): can not find SHOULD_REPLICATE flag, should be hadoop 2.x java.lang.IllegalArgumentException: No enum constant org.apache.hadoop.fs.CreateFlag.SHOULD_REPLICATE at java.lang.Enum.valueOf(Enum.java:238) at org.apache.hadoop.fs.CreateFlag.valueOf(CreateFlag.java:63) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.loadShouldReplicateFlag(FanOutOneBlockAsyncDFSOutputHelper.java:277) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.(FanOutOneBlockAsyncDFSOutputHelper.java:304) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:139) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:53:38,345 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(243): No decryptEncryptedDataEncryptionKey method in DFSClient, should be hadoop version with HDFS-12396 java.lang.NoSuchMethodException: org.apache.hadoop.hdfs.DFSClient.decryptEncryptedDataEncryptionKey(org.apache.hadoop.fs.FileEncryptionInfo) at java.lang.Class.getDeclaredMethod(Class.java:2130) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelperWithoutHDFS12396(FanOutOneBlockAsyncDFSOutputSaslHelper.java:182) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.createTransparentCryptoHelper(FanOutOneBlockAsyncDFSOutputSaslHelper.java:241) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper.(FanOutOneBlockAsyncDFSOutputSaslHelper.java:252) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.wal.AsyncFSWALProvider.load(AsyncFSWALProvider.java:140) at org.apache.hadoop.hbase.wal.WALFactory.getProviderClass(WALFactory.java:135) at org.apache.hadoop.hbase.wal.WALFactory.getProvider(WALFactory.java:175) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:202) at org.apache.hadoop.hbase.wal.WALFactory.(WALFactory.java:182) at org.apache.hadoop.hbase.master.region.MasterRegion.create(MasterRegion.java:339) at org.apache.hadoop.hbase.master.region.MasterRegionFactory.create(MasterRegionFactory.java:104) at org.apache.hadoop.hbase.master.HMaster.finishActiveMasterInitialization(HMaster.java:855) at org.apache.hadoop.hbase.master.HMaster.startActiveMasterManager(HMaster.java:2193) at org.apache.hadoop.hbase.master.HMaster.lambda$run$0(HMaster.java:528) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:53:38,347 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-08 02:53:38,388 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7693): Creating {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='master:store', {NAME => 'proc', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, under table dir hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/MasterData/data/master/store-tmp 2023-04-08 02:53:38,820 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 02:53:38,821 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-08 02:53:38,821 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-08 02:53:38,821 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-08 02:53:38,821 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-08 02:53:38,821 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-08 02:53:38,821 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-08 02:53:38,821 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-08 02:53:38,824 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegion(191): WALDir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/MasterData/WALs/jenkins-hbase4.apache.org,46135,1680922416713 2023-04-08 02:53:38,852 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C46135%2C1680922416713, suffix=, logDir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/MasterData/WALs/jenkins-hbase4.apache.org,46135,1680922416713, archiveDir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/MasterData/oldWALs, maxLogs=10 2023-04-08 02:53:38,926 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36451,DS-ed82e519-2bab-4f8a-8c3b-791831bebafc,DISK] 2023-04-08 02:53:38,936 DEBUG [RS-EventLoopGroup-3-2] asyncfs.ProtobufDecoder(123): Hadoop 3.2 and below use unshaded protobuf. java.lang.ClassNotFoundException: org.apache.hadoop.thirdparty.protobuf.MessageLite at java.net.URLClassLoader.findClass(URLClassLoader.java:387) at java.lang.ClassLoader.loadClass(ClassLoader.java:418) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352) at java.lang.ClassLoader.loadClass(ClassLoader.java:351) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:264) at org.apache.hadoop.hbase.io.asyncfs.ProtobufDecoder.(ProtobufDecoder.java:118) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.processWriteBlockResponse(FanOutOneBlockAsyncDFSOutputHelper.java:340) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$100(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$4.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:424) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.addListener(DefaultPromise.java:185) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.initialize(FanOutOneBlockAsyncDFSOutputHelper.java:418) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper.access$300(FanOutOneBlockAsyncDFSOutputHelper.java:112) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:476) at org.apache.hadoop.hbase.io.asyncfs.FanOutOneBlockAsyncDFSOutputHelper$5.operationComplete(FanOutOneBlockAsyncDFSOutputHelper.java:471) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners0(DefaultPromise.java:583) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:559) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:492) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:636) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:625) at org.apache.hbase.thirdparty.io.netty.util.concurrent.DefaultPromise.trySuccess(DefaultPromise.java:105) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPromise.trySuccess(DefaultChannelPromise.java:84) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.fulfillConnectPromise(AbstractEpollChannel.java:653) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:691) at org.apache.hbase.thirdparty.io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:567) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:489) at org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:397) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:53:39,004 INFO [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/MasterData/WALs/jenkins-hbase4.apache.org,46135,1680922416713/jenkins-hbase4.apache.org%2C46135%2C1680922416713.1680922418867 2023-04-08 02:53:39,005 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36451,DS-ed82e519-2bab-4f8a-8c3b-791831bebafc,DISK]] 2023-04-08 02:53:39,005 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7854): Opening region: {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''} 2023-04-08 02:53:39,006 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(866): Instantiated master:store,,1.1595e783b53d99cd5eef43b6debb2682.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 02:53:39,009 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7894): checking encryption for 1595e783b53d99cd5eef43b6debb2682 2023-04-08 02:53:39,010 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(7897): checking classloading for 1595e783b53d99cd5eef43b6debb2682 2023-04-08 02:53:39,070 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family proc of region 1595e783b53d99cd5eef43b6debb2682 2023-04-08 02:53:39,079 DEBUG [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc 2023-04-08 02:53:39,110 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1595e783b53d99cd5eef43b6debb2682 columnFamilyName proc 2023-04-08 02:53:39,125 INFO [StoreOpener-1595e783b53d99cd5eef43b6debb2682-1] regionserver.HStore(310): Store=1595e783b53d99cd5eef43b6debb2682/proc, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:53:39,131 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-08 02:53:39,133 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682 2023-04-08 02:53:39,152 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1055): writing seq id for 1595e783b53d99cd5eef43b6debb2682 2023-04-08 02:53:39,157 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-08 02:53:39,159 INFO [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(1072): Opened 1595e783b53d99cd5eef43b6debb2682; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73716852, jitterRate=0.09846669435501099}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-08 02:53:39,159 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] regionserver.HRegion(965): Region open journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-08 02:53:39,160 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.MasterRegionFlusherAndCompactor(122): Constructor flushSize=134217728, flushPerChanges=1000000, flushIntervalMs=900000, compactMin=4 2023-04-08 02:53:39,182 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(104): Starting the Region Procedure Store, number threads=5 2023-04-08 02:53:39,183 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(562): Starting 5 core workers (bigger of cpus/4 or 16) with max (burst) worker count=50 2023-04-08 02:53:39,185 INFO [master/jenkins-hbase4:0:becomeActiveMaster] region.RegionProcedureStore(255): Starting Region Procedure Store lease recovery... 2023-04-08 02:53:39,187 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(582): Recovered RegionProcedureStore lease in 1 msec 2023-04-08 02:53:39,221 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(596): Loaded RegionProcedureStore in 33 msec 2023-04-08 02:53:39,221 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.RemoteProcedureDispatcher(95): Instantiated, coreThreads=3 (allowCoreThreadTimeOut=true), queueMaxSize=32, operationDelay=150 2023-04-08 02:53:39,250 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(253): hbase:meta replica znodes: [] 2023-04-08 02:53:39,256 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.RegionServerTracker(124): Starting RegionServerTracker; 0 have existing ServerCrashProcedures, 0 possibly 'live' servers, and 0 'splitting'. 2023-04-08 02:53:39,280 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.BaseLoadBalancer(1082): slop=0.001, systemTablesOnMaster=false 2023-04-08 02:53:39,284 INFO [master/jenkins-hbase4:0:becomeActiveMaster] balancer.StochasticLoadBalancer(253): Loaded config; maxSteps=1000000, runMaxSteps=false, stepsPerRegion=800, maxRunningTime=30000, isByTable=false, CostFunctions=[RegionCountSkewCostFunction, PrimaryRegionCountSkewCostFunction, MoveCostFunction, ServerLocalityCostFunction, RackLocalityCostFunction, TableSkewCostFunction, RegionReplicaHostCostFunction, RegionReplicaRackCostFunction, ReadRequestCostFunction, WriteRequestCostFunction, MemStoreSizeCostFunction, StoreFileCostFunction] , sum of multiplier of cost functions = 0.0 etc. 2023-04-08 02:53:39,286 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/balancer 2023-04-08 02:53:39,291 INFO [master/jenkins-hbase4:0:becomeActiveMaster] normalizer.RegionNormalizerWorker(118): Normalizer rate limit set to unlimited 2023-04-08 02:53:39,295 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/normalizer 2023-04-08 02:53:39,298 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-08 02:53:39,299 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/split 2023-04-08 02:53:39,300 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/switch/merge 2023-04-08 02:53:39,312 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/snapshot-cleanup 2023-04-08 02:53:39,318 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-08 02:53:39,318 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/running 2023-04-08 02:53:39,318 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-08 02:53:39,318 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(744): Active/primary master=jenkins-hbase4.apache.org,46135,1680922416713, sessionid=0x10161b07ca90000, setting cluster-up flag (Was=false) 2023-04-08 02:53:39,334 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-08 02:53:39,339 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/flush-table-proc/acquired, /hbase/flush-table-proc/reached, /hbase/flush-table-proc/abort 2023-04-08 02:53:39,341 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,46135,1680922416713 2023-04-08 02:53:39,346 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-08 02:53:39,352 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureUtil(251): Clearing all znodes /hbase/online-snapshot/acquired, /hbase/online-snapshot/reached, /hbase/online-snapshot/abort 2023-04-08 02:53:39,354 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure.ZKProcedureCoordinator(245): Starting controller for procedure member=jenkins-hbase4.apache.org,46135,1680922416713 2023-04-08 02:53:39,357 WARN [master/jenkins-hbase4:0:becomeActiveMaster] snapshot.SnapshotManager(302): Couldn't delete working snapshot directory: hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.hbase-snapshot/.tmp 2023-04-08 02:53:39,423 INFO [RS:0;jenkins-hbase4:35089] regionserver.HRegionServer(952): ClusterId : 620627a2-e57a-49c8-bcda-753bb9c42bc6 2023-04-08 02:53:39,428 DEBUG [RS:0;jenkins-hbase4:35089] procedure.RegionServerProcedureManagerHost(43): Procedure flush-table-proc initializing 2023-04-08 02:53:39,434 DEBUG [RS:0;jenkins-hbase4:35089] procedure.RegionServerProcedureManagerHost(45): Procedure flush-table-proc initialized 2023-04-08 02:53:39,435 DEBUG [RS:0;jenkins-hbase4:35089] procedure.RegionServerProcedureManagerHost(43): Procedure online-snapshot initializing 2023-04-08 02:53:39,442 DEBUG [RS:0;jenkins-hbase4:35089] procedure.RegionServerProcedureManagerHost(45): Procedure online-snapshot initialized 2023-04-08 02:53:39,443 DEBUG [RS:0;jenkins-hbase4:35089] zookeeper.ReadOnlyZKClient(139): Connect 0x7e8c7836 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:53:39,450 DEBUG [RS:0;jenkins-hbase4:35089] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@704b9997, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:53:39,451 DEBUG [RS:0;jenkins-hbase4:35089] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@74741346, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-08 02:53:39,477 DEBUG [RS:0;jenkins-hbase4:35089] regionserver.ShutdownHook(81): Installed shutdown hook thread: Shutdownhook:RS:0;jenkins-hbase4:35089 2023-04-08 02:53:39,482 INFO [RS:0;jenkins-hbase4:35089] regionserver.RegionServerCoprocessorHost(66): System coprocessor loading is enabled 2023-04-08 02:53:39,482 INFO [RS:0;jenkins-hbase4:35089] regionserver.RegionServerCoprocessorHost(67): Table coprocessor loading is enabled 2023-04-08 02:53:39,482 DEBUG [RS:0;jenkins-hbase4:35089] regionserver.HRegionServer(1023): About to register with Master. 2023-04-08 02:53:39,484 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT; InitMetaProcedure table=hbase:meta 2023-04-08 02:53:39,485 INFO [RS:0;jenkins-hbase4:35089] regionserver.HRegionServer(2810): reportForDuty to master=jenkins-hbase4.apache.org,46135,1680922416713 with isa=jenkins-hbase4.apache.org/172.31.14.131:35089, startcode=1680922417983 2023-04-08 02:53:39,495 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_OPEN_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-08 02:53:39,495 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_CLOSE_REGION-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-08 02:53:39,496 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-08 02:53:39,496 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_META_SERVER_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=5, maxPoolSize=5 2023-04-08 02:53:39,496 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=M_LOG_REPLAY_OPS-master/jenkins-hbase4:0, corePoolSize=10, maxPoolSize=10 2023-04-08 02:53:39,496 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_SNAPSHOT_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-08 02:53:39,496 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_MERGE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-08 02:53:39,496 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] executor.ExecutorService(93): Starting executor service name=MASTER_TABLE_OPERATIONS-master/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-08 02:53:39,500 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.procedure2.CompletedProcedureCleaner; timeout=30000, timestamp=1680922449500 2023-04-08 02:53:39,502 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): log_cleaner Cleaner pool size is 1 2023-04-08 02:53:39,506 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_WRITE_FS_LAYOUT, locked=true; InitMetaProcedure table=hbase:meta 2023-04-08 02:53:39,506 INFO [PEWorker-1] procedure.InitMetaProcedure(71): BOOTSTRAP: creating hbase:meta region 2023-04-08 02:53:39,509 DEBUG [RS:0;jenkins-hbase4:35089] ipc.RpcConnection(124): Using SIMPLE authentication for service=RegionServerStatusService, sasl=false 2023-04-08 02:53:39,512 INFO [PEWorker-1] util.FSTableDescriptors(128): Creating new hbase:meta table descriptor 'hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-08 02:53:39,514 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveLogCleaner 2023-04-08 02:53:39,522 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.replication.master.ReplicationLogCleaner 2023-04-08 02:53:39,523 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreWALCleaner 2023-04-08 02:53:39,523 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveProcedureWALCleaner 2023-04-08 02:53:39,523 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.LogCleaner(148): Creating 1 old WALs cleaner threads 2023-04-08 02:53:39,524 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=LogsCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-08 02:53:39,526 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.DirScanPool(70): hfile_cleaner Cleaner pool size is 2 2023-04-08 02:53:39,528 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveMasterLocalStoreHFileCleaner 2023-04-08 02:53:39,529 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.HFileLinkCleaner 2023-04-08 02:53:39,533 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.snapshot.SnapshotHFileCleaner 2023-04-08 02:53:39,534 INFO [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.CleanerChore(175): Initialize cleaner=org.apache.hadoop.hbase.master.cleaner.TimeToLiveHFileCleaner 2023-04-08 02:53:39,547 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(242): Starting for large file=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680922419540,5,FailOnTimeoutGroup] 2023-04-08 02:53:39,548 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] cleaner.HFileCleaner(257): Starting for small files=Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680922419548,5,FailOnTimeoutGroup] 2023-04-08 02:53:39,549 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HFileCleaner, period=600000, unit=MILLISECONDS is enabled. 2023-04-08 02:53:39,549 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1461): Reopening regions with very high storeFileRefCount is disabled. Provide threshold value > 0 for hbase.regions.recovery.store.file.ref.count to enable it. 2023-04-08 02:53:39,550 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=ReplicationBarrierCleaner, period=43200000, unit=MILLISECONDS is enabled. 2023-04-08 02:53:39,550 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=SnapshotCleaner, period=1800000, unit=MILLISECONDS is enabled. 2023-04-08 02:53:39,576 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52891, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins.hfs.0 (auth:SIMPLE), service=RegionServerStatusService 2023-04-08 02:53:39,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.ServerManager(394): Registering regionserver=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:39,606 DEBUG [RS:0;jenkins-hbase4:35089] regionserver.HRegionServer(1596): Config from master: hbase.rootdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01 2023-04-08 02:53:39,607 DEBUG [RS:0;jenkins-hbase4:35089] regionserver.HRegionServer(1596): Config from master: fs.defaultFS=hdfs://localhost:42901 2023-04-08 02:53:39,607 DEBUG [RS:0;jenkins-hbase4:35089] regionserver.HRegionServer(1596): Config from master: hbase.master.info.port=-1 2023-04-08 02:53:39,611 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-08 02:53:39,612 DEBUG [RS:0;jenkins-hbase4:35089] zookeeper.ZKUtil(162): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:39,612 WARN [RS:0;jenkins-hbase4:35089] hbase.ZNodeClearer(69): Environment variable HBASE_ZNODE_FILE not set; znodes will not be cleared on crash by start scripts (Longer MTTR!) 2023-04-08 02:53:39,613 INFO [RS:0;jenkins-hbase4:35089] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-08 02:53:39,613 DEBUG [RS:0;jenkins-hbase4:35089] regionserver.HRegionServer(1947): logDir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/WALs/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:39,614 INFO [RegionServerTracker-0] master.RegionServerTracker(190): RegionServer ephemeral node created, adding [jenkins-hbase4.apache.org,35089,1680922417983] 2023-04-08 02:53:39,623 DEBUG [RS:0;jenkins-hbase4:35089] zookeeper.ZKUtil(162): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/rs/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:39,635 DEBUG [RS:0;jenkins-hbase4:35089] regionserver.Replication(139): Replication stats-in-log period=300 seconds 2023-04-08 02:53:39,644 INFO [RS:0;jenkins-hbase4:35089] regionserver.MetricsRegionServerWrapperImpl(154): Computing regionserver metrics every 5000 milliseconds 2023-04-08 02:53:39,662 INFO [RS:0;jenkins-hbase4:35089] regionserver.MemStoreFlusher(125): globalMemStoreLimit=782.4 M, globalMemStoreLimitLowMark=743.3 M, Offheap=false 2023-04-08 02:53:39,665 INFO [RS:0;jenkins-hbase4:35089] throttle.PressureAwareCompactionThroughputController(131): Compaction throughput configurations, higher bound: 100.00 MB/second, lower bound 50.00 MB/second, off peak: unlimited, tuning period: 60000 ms 2023-04-08 02:53:39,666 INFO [RS:0;jenkins-hbase4:35089] hbase.ChoreService(166): Chore ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS is enabled. 2023-04-08 02:53:39,667 INFO [RS:0;jenkins-hbase4:35089] regionserver.HRegionServer$CompactionChecker(1838): CompactionChecker runs every PT1S 2023-04-08 02:53:39,673 INFO [RS:0;jenkins-hbase4:35089] hbase.ChoreService(166): Chore ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS is enabled. 2023-04-08 02:53:39,674 DEBUG [RS:0;jenkins-hbase4:35089] executor.ExecutorService(93): Starting executor service name=RS_OPEN_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-08 02:53:39,674 DEBUG [RS:0;jenkins-hbase4:35089] executor.ExecutorService(93): Starting executor service name=RS_OPEN_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-08 02:53:39,674 DEBUG [RS:0;jenkins-hbase4:35089] executor.ExecutorService(93): Starting executor service name=RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-08 02:53:39,674 DEBUG [RS:0;jenkins-hbase4:35089] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_REGION-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-08 02:53:39,675 DEBUG [RS:0;jenkins-hbase4:35089] executor.ExecutorService(93): Starting executor service name=RS_CLOSE_META-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-08 02:53:39,675 DEBUG [RS:0;jenkins-hbase4:35089] executor.ExecutorService(93): Starting executor service name=RS_LOG_REPLAY_OPS-regionserver/jenkins-hbase4:0, corePoolSize=2, maxPoolSize=2 2023-04-08 02:53:39,675 DEBUG [RS:0;jenkins-hbase4:35089] executor.ExecutorService(93): Starting executor service name=RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-08 02:53:39,675 DEBUG [RS:0;jenkins-hbase4:35089] executor.ExecutorService(93): Starting executor service name=RS_REGION_REPLICA_FLUSH_OPS-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-08 02:53:39,675 DEBUG [RS:0;jenkins-hbase4:35089] executor.ExecutorService(93): Starting executor service name=RS_REFRESH_PEER-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-08 02:53:39,676 DEBUG [RS:0;jenkins-hbase4:35089] executor.ExecutorService(93): Starting executor service name=RS_SWITCH_RPC_THROTTLE-regionserver/jenkins-hbase4:0, corePoolSize=1, maxPoolSize=1 2023-04-08 02:53:39,677 INFO [RS:0;jenkins-hbase4:35089] hbase.ChoreService(166): Chore ScheduledChore name=CompactionChecker, period=1000, unit=MILLISECONDS is enabled. 2023-04-08 02:53:39,677 INFO [RS:0;jenkins-hbase4:35089] hbase.ChoreService(166): Chore ScheduledChore name=MemstoreFlusherChore, period=1000, unit=MILLISECONDS is enabled. 2023-04-08 02:53:39,677 INFO [RS:0;jenkins-hbase4:35089] hbase.ChoreService(166): Chore ScheduledChore name=nonceCleaner, period=360000, unit=MILLISECONDS is enabled. 2023-04-08 02:53:39,697 INFO [RS:0;jenkins-hbase4:35089] regionserver.HeapMemoryManager(209): Starting, tuneOn=false 2023-04-08 02:53:39,700 INFO [RS:0;jenkins-hbase4:35089] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,35089,1680922417983-HeapMemoryTunerChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-08 02:53:39,717 INFO [RS:0;jenkins-hbase4:35089] regionserver.Replication(203): jenkins-hbase4.apache.org,35089,1680922417983 started 2023-04-08 02:53:39,717 INFO [RS:0;jenkins-hbase4:35089] regionserver.HRegionServer(1638): Serving as jenkins-hbase4.apache.org,35089,1680922417983, RpcServer on jenkins-hbase4.apache.org/172.31.14.131:35089, sessionid=0x10161b07ca90001 2023-04-08 02:53:39,717 DEBUG [RS:0;jenkins-hbase4:35089] procedure.RegionServerProcedureManagerHost(51): Procedure flush-table-proc starting 2023-04-08 02:53:39,717 DEBUG [RS:0;jenkins-hbase4:35089] flush.RegionServerFlushTableProcedureManager(106): Start region server flush procedure manager jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:39,718 DEBUG [RS:0;jenkins-hbase4:35089] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35089,1680922417983' 2023-04-08 02:53:39,718 DEBUG [RS:0;jenkins-hbase4:35089] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:39,718 DEBUG [RS:0;jenkins-hbase4:35089] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:39,719 DEBUG [RS:0;jenkins-hbase4:35089] procedure.RegionServerProcedureManagerHost(53): Procedure flush-table-proc started 2023-04-08 02:53:39,719 DEBUG [RS:0;jenkins-hbase4:35089] procedure.RegionServerProcedureManagerHost(51): Procedure online-snapshot starting 2023-04-08 02:53:39,719 DEBUG [RS:0;jenkins-hbase4:35089] snapshot.RegionServerSnapshotManager(126): Start Snapshot Manager jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:39,719 DEBUG [RS:0;jenkins-hbase4:35089] procedure.ZKProcedureMemberRpcs(357): Starting procedure member 'jenkins-hbase4.apache.org,35089,1680922417983' 2023-04-08 02:53:39,719 DEBUG [RS:0;jenkins-hbase4:35089] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/online-snapshot/abort' 2023-04-08 02:53:39,719 DEBUG [RS:0;jenkins-hbase4:35089] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/online-snapshot/acquired' 2023-04-08 02:53:39,720 DEBUG [RS:0;jenkins-hbase4:35089] procedure.RegionServerProcedureManagerHost(53): Procedure online-snapshot started 2023-04-08 02:53:39,720 INFO [RS:0;jenkins-hbase4:35089] quotas.RegionServerRpcQuotaManager(63): Quota support disabled 2023-04-08 02:53:39,720 INFO [RS:0;jenkins-hbase4:35089] quotas.RegionServerSpaceQuotaManager(80): Quota support disabled, not starting space quota manager. 2023-04-08 02:53:39,831 INFO [RS:0;jenkins-hbase4:35089] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35089%2C1680922417983, suffix=, logDir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/WALs/jenkins-hbase4.apache.org,35089,1680922417983, archiveDir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/oldWALs, maxLogs=32 2023-04-08 02:53:39,849 DEBUG [RS-EventLoopGroup-3-2] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36451,DS-ed82e519-2bab-4f8a-8c3b-791831bebafc,DISK] 2023-04-08 02:53:39,852 INFO [RS:0;jenkins-hbase4:35089] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/WALs/jenkins-hbase4.apache.org,35089,1680922417983/jenkins-hbase4.apache.org%2C35089%2C1680922417983.1680922419833 2023-04-08 02:53:39,853 DEBUG [RS:0;jenkins-hbase4:35089] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36451,DS-ed82e519-2bab-4f8a-8c3b-791831bebafc,DISK]] 2023-04-08 02:53:39,965 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-08 02:53:39,966 INFO [PEWorker-1] util.FSTableDescriptors(135): Updated hbase:meta table descriptor to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/.tabledesc/.tableinfo.0000000001 2023-04-08 02:53:39,966 INFO [PEWorker-1] regionserver.HRegion(7675): creating {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:meta', {TABLE_ATTRIBUTES => {IS_META => 'true', coprocessor$1 => '|org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint|536870911|'}}, {NAME => 'info', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, {NAME => 'rep_barrier', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '2147483647', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'table', BLOOMFILTER => 'NONE', IN_MEMORY => 'true', VERSIONS => '3', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01 2023-04-08 02:53:40,382 DEBUG [PEWorker-1] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 02:53:40,384 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-08 02:53:40,388 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/1588230740/info 2023-04-08 02:53:40,389 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-08 02:53:40,390 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:53:40,390 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-08 02:53:40,393 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/1588230740/rep_barrier 2023-04-08 02:53:40,393 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-08 02:53:40,394 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:53:40,394 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-08 02:53:40,396 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/1588230740/table 2023-04-08 02:53:40,397 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-08 02:53:40,398 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:53:40,399 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/1588230740 2023-04-08 02:53:40,400 DEBUG [PEWorker-1] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/1588230740 2023-04-08 02:53:40,404 DEBUG [PEWorker-1] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-08 02:53:40,406 DEBUG [PEWorker-1] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-08 02:53:40,409 DEBUG [PEWorker-1] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/1588230740/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-08 02:53:40,410 INFO [PEWorker-1] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65205607, jitterRate=-0.02836073935031891}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-08 02:53:40,410 DEBUG [PEWorker-1] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-08 02:53:40,410 DEBUG [PEWorker-1] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-08 02:53:40,410 INFO [PEWorker-1] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-08 02:53:40,410 DEBUG [PEWorker-1] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-08 02:53:40,410 DEBUG [PEWorker-1] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-08 02:53:40,410 DEBUG [PEWorker-1] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-08 02:53:40,411 INFO [PEWorker-1] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-08 02:53:40,411 DEBUG [PEWorker-1] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-08 02:53:40,416 DEBUG [PEWorker-1] procedure.InitMetaProcedure(92): Execute pid=1, state=RUNNABLE:INIT_META_ASSIGN_META, locked=true; InitMetaProcedure table=hbase:meta 2023-04-08 02:53:40,416 INFO [PEWorker-1] procedure.InitMetaProcedure(103): Going to assign meta 2023-04-08 02:53:40,424 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN}] 2023-04-08 02:53:40,436 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN 2023-04-08 02:53:40,439 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=2, ppid=1, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN; state=OFFLINE, location=null; forceNewPlan=false, retain=false 2023-04-08 02:53:40,591 DEBUG [jenkins-hbase4:46135] assignment.AssignmentManager(2176): Processing assignQueue; systemServersCount=1, allServersCount=1 2023-04-08 02:53:40,595 INFO [PEWorker-4] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,35089,1680922417983, state=OPENING 2023-04-08 02:53:40,603 DEBUG [PEWorker-4] zookeeper.MetaTableLocator(240): hbase:meta region location doesn't exist, create it 2023-04-08 02:53:40,604 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-08 02:53:40,605 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-08 02:53:40,608 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=3, ppid=2, state=RUNNABLE; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,35089,1680922417983}] 2023-04-08 02:53:40,798 DEBUG [RSProcedureDispatcher-pool-0] master.ServerManager(712): New admin connection to jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:40,802 DEBUG [RSProcedureDispatcher-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=AdminService, sasl=false 2023-04-08 02:53:40,805 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52944, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=AdminService 2023-04-08 02:53:40,818 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:meta,,1.1588230740 2023-04-08 02:53:40,818 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.WALFactory(158): Instantiating WALProvider of type class org.apache.hadoop.hbase.wal.AsyncFSWALProvider 2023-04-08 02:53:40,822 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(489): WAL configuration: blocksize=256 MB, rollsize=128 MB, prefix=jenkins-hbase4.apache.org%2C35089%2C1680922417983.meta, suffix=.meta, logDir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/WALs/jenkins-hbase4.apache.org,35089,1680922417983, archiveDir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/oldWALs, maxLogs=32 2023-04-08 02:53:40,839 DEBUG [RS-EventLoopGroup-3-1] asyncfs.FanOutOneBlockAsyncDFSOutputSaslHelper(809): SASL client skipping handshake in unsecured configuration for addr = 127.0.0.1/127.0.0.1, datanodeId = DatanodeInfoWithStorage[127.0.0.1:36451,DS-ed82e519-2bab-4f8a-8c3b-791831bebafc,DISK] 2023-04-08 02:53:40,844 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(806): New WAL /user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/WALs/jenkins-hbase4.apache.org,35089,1680922417983/jenkins-hbase4.apache.org%2C35089%2C1680922417983.meta.1680922420823.meta 2023-04-08 02:53:40,852 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] wal.AbstractFSWAL(887): Create new AsyncFSWAL writer with pipeline: [DatanodeInfoWithStorage[127.0.0.1:36451,DS-ed82e519-2bab-4f8a-8c3b-791831bebafc,DISK]] 2023-04-08 02:53:40,852 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '', ENDKEY => ''} 2023-04-08 02:53:40,854 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(215): Loading coprocessor class org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint with path null and priority 536870911 2023-04-08 02:53:40,871 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(8550): Registered coprocessor service: region=hbase:meta,,1 service=MultiRowMutationService 2023-04-08 02:53:40,879 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.RegionCoprocessorHost(393): Loaded coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint from HTD of hbase:meta successfully. 2023-04-08 02:53:40,884 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table meta 1588230740 2023-04-08 02:53:40,884 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:meta,,1.1588230740; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 02:53:40,884 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 1588230740 2023-04-08 02:53:40,885 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 1588230740 2023-04-08 02:53:40,888 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 1588230740 2023-04-08 02:53:40,890 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/1588230740/info 2023-04-08 02:53:40,890 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/1588230740/info 2023-04-08 02:53:40,890 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName info 2023-04-08 02:53:40,891 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:53:40,891 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family rep_barrier of region 1588230740 2023-04-08 02:53:40,893 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/1588230740/rep_barrier 2023-04-08 02:53:40,893 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/1588230740/rep_barrier 2023-04-08 02:53:40,894 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName rep_barrier 2023-04-08 02:53:40,895 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/rep_barrier, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:53:40,895 INFO [StoreOpener-1588230740-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family table of region 1588230740 2023-04-08 02:53:40,897 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/1588230740/table 2023-04-08 02:53:40,897 DEBUG [StoreOpener-1588230740-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/1588230740/table 2023-04-08 02:53:40,897 INFO [StoreOpener-1588230740-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 1588230740 columnFamilyName table 2023-04-08 02:53:40,898 INFO [StoreOpener-1588230740-1] regionserver.HStore(310): Store=1588230740/table, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:53:40,900 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/1588230740 2023-04-08 02:53:40,904 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/1588230740 2023-04-08 02:53:40,908 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table hbase:meta descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-08 02:53:40,911 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 1588230740 2023-04-08 02:53:40,913 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 1588230740; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60217861, jitterRate=-0.10268394649028778}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-08 02:53:40,913 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 1588230740: 2023-04-08 02:53:40,925 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:meta,,1.1588230740, pid=3, masterSystemTime=1680922420789 2023-04-08 02:53:40,945 DEBUG [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:meta,,1.1588230740 2023-04-08 02:53:40,946 INFO [RS_OPEN_META-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:meta,,1.1588230740 2023-04-08 02:53:40,947 INFO [PEWorker-5] zookeeper.MetaTableLocator(228): Setting hbase:meta replicaId=0 location in ZooKeeper as jenkins-hbase4.apache.org,35089,1680922417983, state=OPEN 2023-04-08 02:53:40,949 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/meta-region-server 2023-04-08 02:53:40,950 DEBUG [zk-event-processor-pool-0] master.MetaRegionLocationCache(164): Updating meta znode for path /hbase/meta-region-server: CHANGED 2023-04-08 02:53:40,954 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=3, resume processing ppid=2 2023-04-08 02:53:40,955 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=3, ppid=2, state=SUCCESS; OpenRegionProcedure 1588230740, server=jenkins-hbase4.apache.org,35089,1680922417983 in 341 msec 2023-04-08 02:53:40,960 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=2, resume processing ppid=1 2023-04-08 02:53:40,961 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=2, ppid=1, state=SUCCESS; TransitRegionStateProcedure table=hbase:meta, region=1588230740, ASSIGN in 532 msec 2023-04-08 02:53:40,966 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=1, state=SUCCESS; InitMetaProcedure table=hbase:meta in 1.5610 sec 2023-04-08 02:53:40,966 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(953): Wait for region servers to report in: status=null, state=RUNNING, startTime=1680922420966, completionTime=-1 2023-04-08 02:53:40,967 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.ServerManager(821): Finished waiting on RegionServer count=1; waited=0ms, expected min=1 server(s), max=1 server(s), master is running 2023-04-08 02:53:40,967 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1517): Joining cluster... 2023-04-08 02:53:41,033 DEBUG [hconnection-0x2d1014ec-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:53:41,036 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52956, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:53:41,054 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1529): Number of RegionServers=1 2023-04-08 02:53:41,054 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$RegionInTransitionChore; timeout=60000, timestamp=1680922481054 2023-04-08 02:53:41,054 INFO [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.TimeoutExecutorThread(81): ADDED pid=-1, state=WAITING_TIMEOUT; org.apache.hadoop.hbase.master.assignment.AssignmentManager$DeadServerMetricRegionChore; timeout=120000, timestamp=1680922541054 2023-04-08 02:53:41,054 INFO [master/jenkins-hbase4:0:becomeActiveMaster] assignment.AssignmentManager(1536): Joined the cluster in 87 msec 2023-04-08 02:53:41,078 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46135,1680922416713-ClusterStatusChore, period=60000, unit=MILLISECONDS is enabled. 2023-04-08 02:53:41,078 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46135,1680922416713-BalancerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-08 02:53:41,078 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46135,1680922416713-RegionNormalizerChore, period=300000, unit=MILLISECONDS is enabled. 2023-04-08 02:53:41,080 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=CatalogJanitor-jenkins-hbase4:46135, period=300000, unit=MILLISECONDS is enabled. 2023-04-08 02:53:41,080 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=HbckChore-, period=3600000, unit=MILLISECONDS is enabled. 2023-04-08 02:53:41,086 DEBUG [master/jenkins-hbase4:0.Chore.1] janitor.CatalogJanitor(175): 2023-04-08 02:53:41,093 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.TableNamespaceManager(92): Namespace table not found. Creating... 2023-04-08 02:53:41,094 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(2148): Client=null/null create 'hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'} 2023-04-08 02:53:41,103 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=hbase:namespace 2023-04-08 02:53:41,106 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_PRE_OPERATION 2023-04-08 02:53:41,108 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-08 02:53:41,130 DEBUG [HFileArchiver-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/hbase/namespace/087fd945a79ca5b739610704cbfa7019 2023-04-08 02:53:41,132 DEBUG [HFileArchiver-1] backup.HFileArchiver(153): Directory hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/hbase/namespace/087fd945a79ca5b739610704cbfa7019 empty. 2023-04-08 02:53:41,133 DEBUG [HFileArchiver-1] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/hbase/namespace/087fd945a79ca5b739610704cbfa7019 2023-04-08 02:53:41,133 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived hbase:namespace regions 2023-04-08 02:53:41,557 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/hbase/namespace/.tabledesc/.tableinfo.0000000001 2023-04-08 02:53:41,559 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(7675): creating {ENCODED => 087fd945a79ca5b739610704cbfa7019, NAME => 'hbase:namespace,,1680922421094.087fd945a79ca5b739610704cbfa7019.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='hbase:namespace', {NAME => 'info', BLOOMFILTER => 'ROW', IN_MEMORY => 'true', VERSIONS => '10', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '8192', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp 2023-04-08 02:53:41,971 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680922421094.087fd945a79ca5b739610704cbfa7019.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 02:53:41,972 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1604): Closing 087fd945a79ca5b739610704cbfa7019, disabling compactions & flushes 2023-04-08 02:53:41,972 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680922421094.087fd945a79ca5b739610704cbfa7019. 2023-04-08 02:53:41,972 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680922421094.087fd945a79ca5b739610704cbfa7019. 2023-04-08 02:53:41,972 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680922421094.087fd945a79ca5b739610704cbfa7019. after waiting 0 ms 2023-04-08 02:53:41,972 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680922421094.087fd945a79ca5b739610704cbfa7019. 2023-04-08 02:53:41,972 INFO [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680922421094.087fd945a79ca5b739610704cbfa7019. 2023-04-08 02:53:41,972 DEBUG [RegionOpenAndInit-hbase:namespace-pool-0] regionserver.HRegion(1558): Region close journal for 087fd945a79ca5b739610704cbfa7019: 2023-04-08 02:53:41,977 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ADD_TO_META 2023-04-08 02:53:41,991 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"hbase:namespace,,1680922421094.087fd945a79ca5b739610704cbfa7019.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680922421980"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680922421980"}]},"ts":"1680922421980"} 2023-04-08 02:53:42,016 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-08 02:53:42,018 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-08 02:53:42,022 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680922422018"}]},"ts":"1680922422018"} 2023-04-08 02:53:42,025 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLING in hbase:meta 2023-04-08 02:53:42,035 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=087fd945a79ca5b739610704cbfa7019, ASSIGN}] 2023-04-08 02:53:42,037 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=hbase:namespace, region=087fd945a79ca5b739610704cbfa7019, ASSIGN 2023-04-08 02:53:42,039 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=5, ppid=4, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=hbase:namespace, region=087fd945a79ca5b739610704cbfa7019, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35089,1680922417983; forceNewPlan=false, retain=false 2023-04-08 02:53:42,190 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=087fd945a79ca5b739610704cbfa7019, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:42,191 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"hbase:namespace,,1680922421094.087fd945a79ca5b739610704cbfa7019.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680922422190"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680922422190"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680922422190"}]},"ts":"1680922422190"} 2023-04-08 02:53:42,197 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=6, ppid=5, state=RUNNABLE; OpenRegionProcedure 087fd945a79ca5b739610704cbfa7019, server=jenkins-hbase4.apache.org,35089,1680922417983}] 2023-04-08 02:53:42,356 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open hbase:namespace,,1680922421094.087fd945a79ca5b739610704cbfa7019. 2023-04-08 02:53:42,357 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 087fd945a79ca5b739610704cbfa7019, NAME => 'hbase:namespace,,1680922421094.087fd945a79ca5b739610704cbfa7019.', STARTKEY => '', ENDKEY => ''} 2023-04-08 02:53:42,358 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table namespace 087fd945a79ca5b739610704cbfa7019 2023-04-08 02:53:42,358 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated hbase:namespace,,1680922421094.087fd945a79ca5b739610704cbfa7019.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 02:53:42,358 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 087fd945a79ca5b739610704cbfa7019 2023-04-08 02:53:42,358 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 087fd945a79ca5b739610704cbfa7019 2023-04-08 02:53:42,360 INFO [StoreOpener-087fd945a79ca5b739610704cbfa7019-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family info of region 087fd945a79ca5b739610704cbfa7019 2023-04-08 02:53:42,362 DEBUG [StoreOpener-087fd945a79ca5b739610704cbfa7019-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/namespace/087fd945a79ca5b739610704cbfa7019/info 2023-04-08 02:53:42,363 DEBUG [StoreOpener-087fd945a79ca5b739610704cbfa7019-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/namespace/087fd945a79ca5b739610704cbfa7019/info 2023-04-08 02:53:42,363 INFO [StoreOpener-087fd945a79ca5b739610704cbfa7019-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 087fd945a79ca5b739610704cbfa7019 columnFamilyName info 2023-04-08 02:53:42,364 INFO [StoreOpener-087fd945a79ca5b739610704cbfa7019-1] regionserver.HStore(310): Store=087fd945a79ca5b739610704cbfa7019/info, memstore type=DefaultMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:53:42,365 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/namespace/087fd945a79ca5b739610704cbfa7019 2023-04-08 02:53:42,366 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/namespace/087fd945a79ca5b739610704cbfa7019 2023-04-08 02:53:42,371 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 087fd945a79ca5b739610704cbfa7019 2023-04-08 02:53:42,374 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/namespace/087fd945a79ca5b739610704cbfa7019/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-08 02:53:42,375 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 087fd945a79ca5b739610704cbfa7019; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64996823, jitterRate=-0.0314718633890152}, FlushLargeStoresPolicy{flushSizeLowerBound=-1} 2023-04-08 02:53:42,375 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 087fd945a79ca5b739610704cbfa7019: 2023-04-08 02:53:42,377 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for hbase:namespace,,1680922421094.087fd945a79ca5b739610704cbfa7019., pid=6, masterSystemTime=1680922422350 2023-04-08 02:53:42,381 DEBUG [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for hbase:namespace,,1680922421094.087fd945a79ca5b739610704cbfa7019. 2023-04-08 02:53:42,381 INFO [RS_OPEN_PRIORITY_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened hbase:namespace,,1680922421094.087fd945a79ca5b739610704cbfa7019. 2023-04-08 02:53:42,383 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=5 updating hbase:meta row=087fd945a79ca5b739610704cbfa7019, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:42,383 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"hbase:namespace,,1680922421094.087fd945a79ca5b739610704cbfa7019.","families":{"info":[{"qualifier":"regioninfo","vlen":41,"tag":[],"timestamp":"1680922422382"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680922422382"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680922422382"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680922422382"}]},"ts":"1680922422382"} 2023-04-08 02:53:42,391 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=6, resume processing ppid=5 2023-04-08 02:53:42,391 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=6, ppid=5, state=SUCCESS; OpenRegionProcedure 087fd945a79ca5b739610704cbfa7019, server=jenkins-hbase4.apache.org,35089,1680922417983 in 190 msec 2023-04-08 02:53:42,394 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=5, resume processing ppid=4 2023-04-08 02:53:42,394 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=5, ppid=4, state=SUCCESS; TransitRegionStateProcedure table=hbase:namespace, region=087fd945a79ca5b739610704cbfa7019, ASSIGN in 356 msec 2023-04-08 02:53:42,395 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-08 02:53:42,396 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"hbase:namespace","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680922422396"}]},"ts":"1680922422396"} 2023-04-08 02:53:42,400 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=hbase:namespace, state=ENABLED in hbase:meta 2023-04-08 02:53:42,403 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=4, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=hbase:namespace execute state=CREATE_TABLE_POST_OPERATION 2023-04-08 02:53:42,406 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=4, state=SUCCESS; CreateTableProcedure table=hbase:namespace in 1.3090 sec 2023-04-08 02:53:42,407 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/namespace 2023-04-08 02:53:42,413 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/namespace 2023-04-08 02:53:42,414 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-08 02:53:42,449 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=7, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=default 2023-04-08 02:53:42,463 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-08 02:53:42,469 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=7, state=SUCCESS; CreateNamespaceProcedure, namespace=default in 27 msec 2023-04-08 02:53:42,472 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] procedure2.ProcedureExecutor(1029): Stored pid=8, state=RUNNABLE:CREATE_NAMESPACE_PREPARE; CreateNamespaceProcedure, namespace=hbase 2023-04-08 02:53:42,483 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/namespace 2023-04-08 02:53:42,488 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=8, state=SUCCESS; CreateNamespaceProcedure, namespace=hbase in 15 msec 2023-04-08 02:53:42,497 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/default 2023-04-08 02:53:42,500 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDataChanged, state=SyncConnected, path=/hbase/namespace/hbase 2023-04-08 02:53:42,500 INFO [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1083): Master has completed initialization 4.441sec 2023-04-08 02:53:42,502 INFO [master/jenkins-hbase4:0:becomeActiveMaster] quotas.MasterQuotaManager(97): Quota support disabled 2023-04-08 02:53:42,504 INFO [master/jenkins-hbase4:0:becomeActiveMaster] slowlog.SlowLogMasterService(57): Slow/Large requests logging to system table hbase:slowlog is disabled. Quitting. 2023-04-08 02:53:42,504 INFO [master/jenkins-hbase4:0:becomeActiveMaster] zookeeper.ZKWatcher(269): not a secure deployment, proceeding 2023-04-08 02:53:42,505 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46135,1680922416713-ExpiredMobFileCleanerChore, period=86400, unit=SECONDS is enabled. 2023-04-08 02:53:42,506 INFO [master/jenkins-hbase4:0:becomeActiveMaster] hbase.ChoreService(166): Chore ScheduledChore name=jenkins-hbase4.apache.org,46135,1680922416713-MobCompactionChore, period=604800, unit=SECONDS is enabled. 2023-04-08 02:53:42,516 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster] master.HMaster(1175): Balancer post startup initialization complete, took 0 seconds 2023-04-08 02:53:42,529 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x6ed65498 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:53:42,533 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@f3002be, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:53:42,545 DEBUG [hconnection-0x14c3948e-shared-pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:53:42,556 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:52972, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:53:42,568 INFO [Listener at localhost/39741] hbase.HBaseTestingUtility(1145): Minicluster is up; activeMaster=jenkins-hbase4.apache.org,46135,1680922416713 2023-04-08 02:53:42,592 INFO [Listener at localhost/39741] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=207, OpenFileDescriptor=470, MaxFileDescriptor=60000, SystemLoadAverage=336, ProcessCount=171, AvailableMemoryMB=2356 2023-04-08 02:53:42,609 DEBUG [Listener at localhost/39741] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-08 02:53:42,612 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58380, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-08 02:53:42,621 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-08 02:53:42,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-08 02:53:42,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure2.ProcedureExecutor(1029): Stored pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-08 02:53:42,629 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-08 02:53:42,631 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-08 02:53:42,634 DEBUG [HFileArchiver-2] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:42,634 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 9 2023-04-08 02:53:42,635 DEBUG [HFileArchiver-2] backup.HFileArchiver(153): Directory hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415 empty. 2023-04-08 02:53:42,636 DEBUG [HFileArchiver-2] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:42,637 DEBUG [PEWorker-2] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-08 02:53:42,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-08 02:53:42,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-08 02:53:42,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-08 02:53:43,053 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-08 02:53:43,056 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 9a2d8d09b1a5f9cc567f207bd50b4415, NAME => 'TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp 2023-04-08 02:53:43,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-08 02:53:43,469 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 02:53:43,469 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 9a2d8d09b1a5f9cc567f207bd50b4415, disabling compactions & flushes 2023-04-08 02:53:43,469 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:43,469 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:43,469 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. after waiting 0 ms 2023-04-08 02:53:43,470 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:43,470 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:43,470 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:43,474 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-08 02:53:43,476 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922423476"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680922423476"}]},"ts":"1680922423476"} 2023-04-08 02:53:43,479 INFO [PEWorker-2] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-08 02:53:43,480 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-08 02:53:43,480 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680922423480"}]},"ts":"1680922423480"} 2023-04-08 02:53:43,483 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-08 02:53:43,487 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9a2d8d09b1a5f9cc567f207bd50b4415, ASSIGN}] 2023-04-08 02:53:43,489 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9a2d8d09b1a5f9cc567f207bd50b4415, ASSIGN 2023-04-08 02:53:43,491 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=10, ppid=9, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=9a2d8d09b1a5f9cc567f207bd50b4415, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35089,1680922417983; forceNewPlan=false, retain=false 2023-04-08 02:53:43,642 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=9a2d8d09b1a5f9cc567f207bd50b4415, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:43,642 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922423642"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680922423642"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680922423642"}]},"ts":"1680922423642"} 2023-04-08 02:53:43,646 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=11, ppid=10, state=RUNNABLE; OpenRegionProcedure 9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983}] 2023-04-08 02:53:43,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-08 02:53:43,804 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:43,804 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 9a2d8d09b1a5f9cc567f207bd50b4415, NAME => 'TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.', STARTKEY => '', ENDKEY => ''} 2023-04-08 02:53:43,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:43,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 02:53:43,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:43,805 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:43,807 INFO [StoreOpener-9a2d8d09b1a5f9cc567f207bd50b4415-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:43,809 DEBUG [StoreOpener-9a2d8d09b1a5f9cc567f207bd50b4415-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A 2023-04-08 02:53:43,810 DEBUG [StoreOpener-9a2d8d09b1a5f9cc567f207bd50b4415-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A 2023-04-08 02:53:43,811 INFO [StoreOpener-9a2d8d09b1a5f9cc567f207bd50b4415-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 02:53:43,811 INFO [StoreOpener-9a2d8d09b1a5f9cc567f207bd50b4415-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9a2d8d09b1a5f9cc567f207bd50b4415 columnFamilyName A 2023-04-08 02:53:43,812 INFO [StoreOpener-9a2d8d09b1a5f9cc567f207bd50b4415-1] regionserver.HStore(310): Store=9a2d8d09b1a5f9cc567f207bd50b4415/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:53:43,812 INFO [StoreOpener-9a2d8d09b1a5f9cc567f207bd50b4415-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:43,815 DEBUG [StoreOpener-9a2d8d09b1a5f9cc567f207bd50b4415-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B 2023-04-08 02:53:43,815 DEBUG [StoreOpener-9a2d8d09b1a5f9cc567f207bd50b4415-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B 2023-04-08 02:53:43,815 INFO [StoreOpener-9a2d8d09b1a5f9cc567f207bd50b4415-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 02:53:43,815 INFO [StoreOpener-9a2d8d09b1a5f9cc567f207bd50b4415-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9a2d8d09b1a5f9cc567f207bd50b4415 columnFamilyName B 2023-04-08 02:53:43,816 INFO [StoreOpener-9a2d8d09b1a5f9cc567f207bd50b4415-1] regionserver.HStore(310): Store=9a2d8d09b1a5f9cc567f207bd50b4415/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:53:43,816 INFO [StoreOpener-9a2d8d09b1a5f9cc567f207bd50b4415-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:43,819 DEBUG [StoreOpener-9a2d8d09b1a5f9cc567f207bd50b4415-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C 2023-04-08 02:53:43,819 DEBUG [StoreOpener-9a2d8d09b1a5f9cc567f207bd50b4415-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C 2023-04-08 02:53:43,819 INFO [StoreOpener-9a2d8d09b1a5f9cc567f207bd50b4415-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 02:53:43,819 INFO [StoreOpener-9a2d8d09b1a5f9cc567f207bd50b4415-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 9a2d8d09b1a5f9cc567f207bd50b4415 columnFamilyName C 2023-04-08 02:53:43,820 INFO [StoreOpener-9a2d8d09b1a5f9cc567f207bd50b4415-1] regionserver.HStore(310): Store=9a2d8d09b1a5f9cc567f207bd50b4415/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:53:43,821 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:43,823 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:43,823 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:43,826 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-08 02:53:43,829 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:43,832 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-08 02:53:43,833 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 9a2d8d09b1a5f9cc567f207bd50b4415; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=66558818, jitterRate=-0.008196324110031128}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-08 02:53:43,833 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:43,834 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., pid=11, masterSystemTime=1680922423798 2023-04-08 02:53:43,837 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:43,837 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:43,838 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=10 updating hbase:meta row=9a2d8d09b1a5f9cc567f207bd50b4415, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:43,838 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922423838"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680922423838"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680922423838"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680922423838"}]},"ts":"1680922423838"} 2023-04-08 02:53:43,843 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=11, resume processing ppid=10 2023-04-08 02:53:43,843 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=11, ppid=10, state=SUCCESS; OpenRegionProcedure 9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 in 194 msec 2023-04-08 02:53:43,846 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=10, resume processing ppid=9 2023-04-08 02:53:43,846 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=10, ppid=9, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9a2d8d09b1a5f9cc567f207bd50b4415, ASSIGN in 356 msec 2023-04-08 02:53:43,847 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-08 02:53:43,848 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680922423847"}]},"ts":"1680922423847"} 2023-04-08 02:53:43,850 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-08 02:53:43,853 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=9, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-08 02:53:43,856 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=9, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.2280 sec 2023-04-08 02:53:44,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=9 2023-04-08 02:53:44,758 INFO [Listener at localhost/39741] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 9 completed 2023-04-08 02:53:44,763 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x08c3f619 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:53:44,768 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@b17e1c1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:53:44,771 DEBUG [Listener at localhost/39741] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:53:44,774 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39426, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:53:44,777 DEBUG [Listener at localhost/39741] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-08 02:53:44,780 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:36776, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-08 02:53:44,789 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x748f869b to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:53:44,793 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@56c8e45e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:53:44,793 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x56223d42 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:53:44,798 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4fea27db, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:53:44,798 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x30eb7fb2 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:53:44,803 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6038a026, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:53:44,804 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x2de4d1f6 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:53:44,811 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@63859c62, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:53:44,811 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x02c57e1b to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:53:44,822 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@dbb9ac1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:53:44,824 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x49d71120 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:53:44,830 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6fef7a21, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:53:44,831 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x59d6b6ea to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:53:44,878 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2879e2e1, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:53:44,879 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x0d24bbb7 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:53:44,883 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6aad2e18, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:53:44,884 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x63871a9d to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:53:44,888 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4bd6fc59, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:53:44,894 DEBUG [hconnection-0xfa92e03-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:53:44,895 DEBUG [hconnection-0x544eb923-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:53:44,896 DEBUG [hconnection-0x26f80c4e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:53:44,896 DEBUG [hconnection-0x7c8e030b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:53:44,898 DEBUG [hconnection-0x527b0319-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:53:44,900 DEBUG [hconnection-0x1e045d2b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:53:44,900 DEBUG [hconnection-0x5b5f7a77-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:53:44,900 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39438, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:53:44,901 DEBUG [hconnection-0x7576e4ed-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:53:44,901 DEBUG [hconnection-0x55d4bc5a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:53:44,906 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39434, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:53:44,906 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39442, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:53:44,907 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39446, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:53:44,908 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39458, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:53:44,910 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39482, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:53:44,911 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39470, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:53:44,912 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39488, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:53:44,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:44,925 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:39498, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:53:44,938 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:44,947 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:44,947 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:44,952 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:44,952 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:44,953 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:44,953 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:44,955 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:44,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:44,955 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:44,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:44,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:44,957 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:44,957 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:44,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:44,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:44,958 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:44,959 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:44,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:44,961 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:44,961 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:44,962 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:44,963 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:44,963 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:44,964 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:44,965 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:44,968 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-08 02:53:44,986 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:53:44,987 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:44,987 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:53:44,988 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:44,988 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:53:44,988 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:45,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:45,009 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:45,148 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:45,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922485120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:45,150 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:45,150 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922485135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:45,154 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:45,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39438 deadline: 1680922485135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:45,155 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:45,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39470 deadline: 1680922485139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:45,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680922425065, max timestamp=1680922425118Num uniques 859; 2023-04-08 02:53:45,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680922425065, max timestamp=1680922425094Num uniques 859; 2023-04-08 02:53:45,160 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-08 02:53:45,160 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-08 02:53:45,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680922425065, max timestamp=1680922425118Num uniques 859; 2023-04-08 02:53:45,185 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-08 02:53:45,283 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680922425245, max timestamp=1680922425263Num uniques 1009; 2023-04-08 02:53:45,283 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-08 02:53:45,291 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680922425245, max timestamp=1680922425263Num uniques 1009; 2023-04-08 02:53:45,292 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=7688, suffix heap size=132904,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=168 2023-04-08 02:53:45,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680922425245, max timestamp=1680922425263Num uniques 1009; 2023-04-08 02:53:45,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46213, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-08 02:53:45,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=227, cellSize=10392, totalHeapSize=23848, min timestamp=1680922425319, max timestamp=1680922425327Num uniques 1027; 2023-04-08 02:53:45,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680922425319, max timestamp=1680922425327Num uniques 1009; 2023-04-08 02:53:45,345 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=6870, suffix heap size=132904,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=150 2023-04-08 02:53:45,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680922425319, max timestamp=1680922425327Num uniques 1027; 2023-04-08 02:53:45,347 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=6870, suffix heap size=134776,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=150 2023-04-08 02:53:45,350 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=10392, suffix heap size=134776,new segment heap size=23848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=227 2023-04-08 02:53:45,416 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:45,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922485373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:45,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680922425368, max timestamp=1680922425404Num uniques 1009; 2023-04-08 02:53:45,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680922425368, max timestamp=1680922425404Num uniques 1009; 2023-04-08 02:53:45,417 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8516, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-04-08 02:53:45,418 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=8516, suffix heap size=132904,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=186 2023-04-08 02:53:45,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8510, totalHeapSize=19584, min timestamp=1680922425368, max timestamp=1680922425406Num uniques 1086; 2023-04-08 02:53:45,419 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49730, new segment data size=8510, suffix heap size=140912,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1086, new segment cells count=186 2023-04-08 02:53:45,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680922425440, max timestamp=1680922425449Num uniques 1045; 2023-04-08 02:53:45,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680922425440, max timestamp=1680922425449Num uniques 1045; 2023-04-08 02:53:45,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680922425440, max timestamp=1680922425449Num uniques 1045; 2023-04-08 02:53:45,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-08 02:53:45,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-08 02:53:45,455 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-08 02:53:45,487 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680922425474, max timestamp=1680922425484Num uniques 1054; 2023-04-08 02:53:45,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-08 02:53:45,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680922425474, max timestamp=1680922425483Num uniques 1054; 2023-04-08 02:53:45,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680922425474, max timestamp=1680922425484Num uniques 1054; 2023-04-08 02:53:45,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-08 02:53:45,488 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-08 02:53:45,518 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1680922425484, max timestamp=1680922425515Num uniques 1063; 2023-04-08 02:53:45,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-08 02:53:45,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=263, cellSize=12038, totalHeapSize=27592, min timestamp=1680922425484, max timestamp=1680922425515Num uniques 1063; 2023-04-08 02:53:45,519 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=12038, suffix heap size=138520,new segment heap size=27592 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=263 2023-04-08 02:53:45,523 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680922425509, max timestamp=1680922425515Num uniques 1063; 2023-04-08 02:53:45,524 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-08 02:53:45,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:45,572 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:45,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922485564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:45,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922485564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:45,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:45,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39438 deadline: 1680922485570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:45,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680922425553, max timestamp=1680922425569Num uniques 1122; 2023-04-08 02:53:45,575 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=6870, suffix heap size=144656,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=150 2023-04-08 02:53:45,575 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:45,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39442 deadline: 1680922485573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:45,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680922425553, max timestamp=1680922425569Num uniques 1122; 2023-04-08 02:53:45,577 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=51382, new segment data size=6870, suffix heap size=144656,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1122, new segment cells count=150 2023-04-08 02:53:45,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680922425553, max timestamp=1680922425569Num uniques 1022; 2023-04-08 02:53:45,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=6870, suffix heap size=134256,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=150 2023-04-08 02:53:45,586 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/c496f25dff044743a66f1e06131fa6dc 2023-04-08 02:53:45,642 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'TestAcidGuarantees' 2023-04-08 02:53:45,645 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:meta' 2023-04-08 02:53:45,647 DEBUG [HBase-Metrics2-1] regionserver.MetricsTableSourceImpl(130): Creating new MetricsTableSourceImpl for table 'hbase:namespace' 2023-04-08 02:53:45,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680922425649, max timestamp=1680922425681Num uniques 1009; 2023-04-08 02:53:45,685 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=12866, suffix heap size=132904,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=281 2023-04-08 02:53:45,686 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680922425649, max timestamp=1680922425681Num uniques 1009; 2023-04-08 02:53:45,686 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=281, cellSize=12866, totalHeapSize=29464, min timestamp=1680922425649, max timestamp=1680922425681Num uniques 1009; 2023-04-08 02:53:45,686 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=12866, suffix heap size=132904,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=281 2023-04-08 02:53:45,686 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46214, new segment data size=12866, suffix heap size=132904,new segment heap size=29464 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1009, new segment cells count=281 2023-04-08 02:53:45,722 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:45,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680922425702, max timestamp=1680922425713Num uniques 1140; 2023-04-08 02:53:45,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922485710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:45,722 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=10990, suffix heap size=146528,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=240 2023-04-08 02:53:45,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680922425702, max timestamp=1680922425713Num uniques 1140; 2023-04-08 02:53:45,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=10990, suffix heap size=146528,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=240 2023-04-08 02:53:45,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680922425702, max timestamp=1680922425713Num uniques 1140; 2023-04-08 02:53:45,724 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52210, new segment data size=10990, suffix heap size=146528,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1140, new segment cells count=240 2023-04-08 02:53:45,761 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680922425737, max timestamp=1680922425757Num uniques 1099; 2023-04-08 02:53:45,761 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-08 02:53:45,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680922425737, max timestamp=1680922425758Num uniques 1099; 2023-04-08 02:53:45,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-08 02:53:45,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9104, totalHeapSize=20936, min timestamp=1680922425737, max timestamp=1680922425758Num uniques 1099; 2023-04-08 02:53:45,762 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50324, new segment data size=9104, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-08 02:53:45,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9518, totalHeapSize=21872, min timestamp=1680922425816, max timestamp=1680922425827Num uniques 1058; 2023-04-08 02:53:45,830 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48448, new segment data size=9518, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-08 02:53:45,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9518, totalHeapSize=21872, min timestamp=1680922425816, max timestamp=1680922425827Num uniques 1058; 2023-04-08 02:53:45,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48448, new segment data size=9518, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-08 02:53:45,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680922425816, max timestamp=1680922425827Num uniques 1058; 2023-04-08 02:53:45,831 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=9520, suffix heap size=138000,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=208 2023-04-08 02:53:45,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1680922425856, max timestamp=1680922425865Num uniques 1067; 2023-04-08 02:53:45,869 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-08 02:53:45,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1680922425856, max timestamp=1680922425865Num uniques 1067; 2023-04-08 02:53:45,870 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-08 02:53:45,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=267, cellSize=12222, totalHeapSize=28008, min timestamp=1680922425856, max timestamp=1680922425865Num uniques 1067; 2023-04-08 02:53:45,872 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=12222, suffix heap size=138936,new segment heap size=28008 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=267 2023-04-08 02:53:45,964 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:45,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39438 deadline: 1680922485956, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:45,965 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:45,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:45,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39442 deadline: 1680922485957, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:45,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922485959, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:45,967 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:45,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39470 deadline: 1680922485965, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:45,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:45,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922485967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,068 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:46,069 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:46,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39442 deadline: 1680922486068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39438 deadline: 1680922486069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:46,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922486070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:46,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922486072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,074 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:46,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39470 deadline: 1680922486072, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,141 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/90730644cd40498e8aad1e01bd6a3c37 2023-04-08 02:53:46,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:46,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39442 deadline: 1680922486276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,279 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:46,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922486277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:46,280 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:46,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39438 deadline: 1680922486277, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922486278, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:46,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39470 deadline: 1680922486286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,433 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-08 02:53:46,581 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:46,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39442 deadline: 1680922486579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,582 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:46,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922486581, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,585 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:46,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922486585, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,592 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:46,592 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:46,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39438 deadline: 1680922486584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39470 deadline: 1680922486589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,621 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=9 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/17d3302666834341a335679dc15f7e10 2023-04-08 02:53:46,636 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/c496f25dff044743a66f1e06131fa6dc as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/c496f25dff044743a66f1e06131fa6dc 2023-04-08 02:53:46,653 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/c496f25dff044743a66f1e06131fa6dc, entries=100, sequenceid=9, filesize=9.3 K 2023-04-08 02:53:46,661 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/90730644cd40498e8aad1e01bd6a3c37 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/90730644cd40498e8aad1e01bd6a3c37 2023-04-08 02:53:46,685 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/90730644cd40498e8aad1e01bd6a3c37, entries=100, sequenceid=9, filesize=9.3 K 2023-04-08 02:53:46,689 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/17d3302666834341a335679dc15f7e10 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/17d3302666834341a335679dc15f7e10 2023-04-08 02:53:46,699 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/17d3302666834341a335679dc15f7e10, entries=100, sequenceid=9, filesize=9.3 K 2023-04-08 02:53:46,708 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=147.60 KB/151140 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 1740ms, sequenceid=9, compaction requested=false 2023-04-08 02:53:46,709 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:46,709 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:46,709 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:46,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:46,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:46,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:46,713 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:46,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:46,714 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:46,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:46,715 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:46,715 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:46,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:46,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:46,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:46,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:46,718 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11c9a2a2[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:46,718 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:46,718 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:46,720 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:46,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:46,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:46,720 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:46,720 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:46,720 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:46,720 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,720 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:46,725 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:46,725 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,725 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:46,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:46,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:46,725 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:46,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:46,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:46,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:46,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:46,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:46,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:46,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:46,733 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:46,733 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:46,733 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:46,733 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:46,733 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:46,736 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:46,736 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:46,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:46,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:46,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:46,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:46,738 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:46,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:46,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:46,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:46,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:46,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:46,740 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:46,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:46,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:46,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,744 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:46,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:46,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,758 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:46,758 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:46,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:46,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:46,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:46,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:46,760 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,760 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:46,760 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:46,760 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,760 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:46,760 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:46,760 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:46,768 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:46,768 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:46,770 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:46,770 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:46,770 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:46,871 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:46,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:46,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:46,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:46,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:46,895 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:46,895 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:46,897 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:46,897 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:46,898 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:46,898 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:46,901 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:46,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:46,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:46,902 DEBUG [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(130): Registering adapter for the MetricRegistry: RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-08 02:53:46,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:46,903 INFO [HBase-Metrics2-1] impl.GlobalMetricRegistriesAdapter(134): Registering RegionServer,sub=Coprocessor.Region.CP_org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint Metrics about HBase RegionObservers 2023-04-08 02:53:46,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:46,904 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:46,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:46,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:46,905 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:46,905 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:46,912 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:46,912 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:46,912 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:46,912 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:46,913 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:46,914 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:46,914 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:46,914 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:46,914 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:46,914 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=147.60 KB heapSize=414.70 KB 2023-04-08 02:53:46,915 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:53:46,916 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:53:46,916 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:53:46,916 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:53:46,916 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:53:46,916 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:53:47,000 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/3f59fb0598cd4e448ab979202798dbbc 2023-04-08 02:53:47,065 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/e9a1795d40e74cf5bcc889cbaf98c034 2023-04-08 02:53:47,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:47,120 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:47,128 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=252 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/05aa5b6366f94252907e1836ac22f4f1 2023-04-08 02:53:47,135 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:47,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39442 deadline: 1680922487121, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,136 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:47,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922487122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,137 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:47,137 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922487128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,140 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:47,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39470 deadline: 1680922487130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,141 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:47,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39438 deadline: 1680922487131, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,144 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/3f59fb0598cd4e448ab979202798dbbc as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/3f59fb0598cd4e448ab979202798dbbc 2023-04-08 02:53:47,156 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/3f59fb0598cd4e448ab979202798dbbc, entries=150, sequenceid=252, filesize=11.8 K 2023-04-08 02:53:47,158 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/e9a1795d40e74cf5bcc889cbaf98c034 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/e9a1795d40e74cf5bcc889cbaf98c034 2023-04-08 02:53:47,176 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/e9a1795d40e74cf5bcc889cbaf98c034, entries=150, sequenceid=252, filesize=11.8 K 2023-04-08 02:53:47,178 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/05aa5b6366f94252907e1836ac22f4f1 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/05aa5b6366f94252907e1836ac22f4f1 2023-04-08 02:53:47,198 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/05aa5b6366f94252907e1836ac22f4f1, entries=150, sequenceid=252, filesize=11.8 K 2023-04-08 02:53:47,201 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~147.60 KB/151140, heapSize ~414.66 KB/424608, currentSize=33.54 KB/34350 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 287ms, sequenceid=252, compaction requested=false 2023-04-08 02:53:47,201 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:47,201 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:47,201 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:47,202 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:47,202 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,202 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:47,202 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:47,213 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:47,213 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:47,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:47,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:47,214 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:47,214 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:47,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:47,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:47,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:47,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:47,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5284adbd[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:47,217 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:47,217 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:47,219 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:47,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:47,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:47,220 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:47,220 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,220 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:47,220 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:47,220 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:47,229 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:47,229 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:47,229 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:47,230 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:47,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:47,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:47,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:47,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:47,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:47,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:47,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:47,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:47,247 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:47,247 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:47,247 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:47,247 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:47,247 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:47,253 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:47,253 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:47,253 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:47,253 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:47,253 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:47,253 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-08 02:53:47,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:47,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:47,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:47,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:47,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:47,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:47,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:47,255 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:47,256 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:53:47,256 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:47,256 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:53:47,256 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:47,256 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:53:47,256 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:47,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:47,259 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:47,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:47,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:47,283 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:47,283 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:47,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:47,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:47,284 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:47,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:47,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:47,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:47,284 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:47,284 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:47,285 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:47,302 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:47,302 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,303 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:47,302 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:47,303 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:47,303 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:47,303 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:47,363 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:47,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922487336, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,366 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:47,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39470 deadline: 1680922487337, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:47,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39438 deadline: 1680922487339, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,367 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:47,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922487340, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,369 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/d3180eb0a9094c04953acf896f907f20 2023-04-08 02:53:47,379 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:47,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39442 deadline: 1680922487366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,403 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:47,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:47,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:47,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:47,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:47,429 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:47,429 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:47,430 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:47,430 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:47,431 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:47,431 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:47,433 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,433 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:47,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:47,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:47,434 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/dd985b26bff540cc98b7d1610b82fc85 2023-04-08 02:53:47,435 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,435 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:47,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:47,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:47,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:47,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:47,441 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:47,442 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:47,442 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:47,442 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:47,442 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:47,443 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:47,444 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:47,444 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:47,444 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:47,445 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:47,445 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:47,445 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:47,446 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:47,446 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:47,446 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,446 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:47,446 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:47,451 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:47,451 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:47,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:47,452 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:47,452 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:47,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:47,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:47,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:47,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:47,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b8a018d[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:47,455 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:47,455 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:47,457 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:47,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:47,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:47,458 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:47,458 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:47,458 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:47,458 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,459 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:47,465 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:47,465 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,465 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:47,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:47,465 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:47,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:47,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:47,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:47,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:47,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:47,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:47,472 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:47,472 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:47,472 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:47,473 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:47,473 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:47,474 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:47,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922487469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,475 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:47,476 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:47,476 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:47,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:47,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:47,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:47,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:47,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:47,476 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:47,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:47,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:47,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:47,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:47,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:47,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:47,480 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:47,484 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:47,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39470 deadline: 1680922487473, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:47,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922487474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,485 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:47,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:47,486 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39438 deadline: 1680922487474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,488 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:47,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39442 deadline: 1680922487484, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,517 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:47,518 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:47,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:47,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:47,519 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,519 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:47,519 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:47,519 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,519 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:47,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:47,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:47,519 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:47,519 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:47,519 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:47,519 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:47,519 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:47,520 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:47,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:47,523 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=265 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/e996d940d4cd49babd94a6f624810b59 2023-04-08 02:53:47,536 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/d3180eb0a9094c04953acf896f907f20 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/d3180eb0a9094c04953acf896f907f20 2023-04-08 02:53:47,546 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/d3180eb0a9094c04953acf896f907f20, entries=350, sequenceid=265, filesize=21.4 K 2023-04-08 02:53:47,548 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/dd985b26bff540cc98b7d1610b82fc85 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/dd985b26bff540cc98b7d1610b82fc85 2023-04-08 02:53:47,559 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/dd985b26bff540cc98b7d1610b82fc85, entries=150, sequenceid=265, filesize=11.9 K 2023-04-08 02:53:47,562 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/e996d940d4cd49babd94a6f624810b59 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/e996d940d4cd49babd94a6f624810b59 2023-04-08 02:53:47,574 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/e996d940d4cd49babd94a6f624810b59, entries=150, sequenceid=265, filesize=11.9 K 2023-04-08 02:53:47,578 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 325ms, sequenceid=265, compaction requested=true 2023-04-08 02:53:47,578 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:47,585 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:53:47,585 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:53:47,585 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:53:47,586 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:53:47,586 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:53:47,592 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33764 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:53:47,592 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 43478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:53:47,595 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/B is initiating minor compaction (all files) 2023-04-08 02:53:47,595 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/A is initiating minor compaction (all files) 2023-04-08 02:53:47,595 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/B in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:47,595 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/A in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:47,595 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/c496f25dff044743a66f1e06131fa6dc, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/3f59fb0598cd4e448ab979202798dbbc, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/d3180eb0a9094c04953acf896f907f20] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=42.5 K 2023-04-08 02:53:47,595 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/90730644cd40498e8aad1e01bd6a3c37, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/e9a1795d40e74cf5bcc889cbaf98c034, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/dd985b26bff540cc98b7d1610b82fc85] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=33.0 K 2023-04-08 02:53:47,597 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 90730644cd40498e8aad1e01bd6a3c37, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1680922424925 2023-04-08 02:53:47,597 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting c496f25dff044743a66f1e06131fa6dc, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1680922424925 2023-04-08 02:53:47,599 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting e9a1795d40e74cf5bcc889cbaf98c034, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1680922425955 2023-04-08 02:53:47,599 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 3f59fb0598cd4e448ab979202798dbbc, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1680922425955 2023-04-08 02:53:47,600 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting d3180eb0a9094c04953acf896f907f20, keycount=350, bloomtype=ROW, size=21.4 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1680922427120 2023-04-08 02:53:47,600 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting dd985b26bff540cc98b7d1610b82fc85, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1680922427251 2023-04-08 02:53:47,620 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:47,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:47,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:47,630 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#B#compaction#9 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:53:47,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:47,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:47,643 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:47,643 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:47,644 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:47,644 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:47,645 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:47,645 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:47,648 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#A#compaction#10 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:53:47,653 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,653 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:47,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:47,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:47,655 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:47,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:47,655 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:47,656 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,656 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:47,663 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:47,663 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:47,668 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:47,670 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:47,671 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:47,672 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:47,672 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:47,672 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:47,672 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:47,672 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-08 02:53:47,675 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:53:47,675 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:47,675 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:53:47,675 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:47,675 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:53:47,676 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:47,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:47,689 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:47,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:47,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922487699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,716 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:47,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39438 deadline: 1680922487709, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,723 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:47,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922487713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,729 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:47,729 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39442 deadline: 1680922487716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,730 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:47,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39470 deadline: 1680922487716, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,732 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/d39bdab3821d4346b3be7e6a8517746c 2023-04-08 02:53:47,736 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/17d35ecf0f8e4f73913d8069fb96909a as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/17d35ecf0f8e4f73913d8069fb96909a 2023-04-08 02:53:47,776 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/B of 9a2d8d09b1a5f9cc567f207bd50b4415 into 17d35ecf0f8e4f73913d8069fb96909a(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:53:47,776 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:47,776 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/B, priority=13, startTime=1680922427585; duration=0sec 2023-04-08 02:53:47,777 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:53:47,777 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:53:47,779 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 33764 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:53:47,779 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/C is initiating minor compaction (all files) 2023-04-08 02:53:47,779 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/C in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:47,779 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/17d3302666834341a335679dc15f7e10, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/05aa5b6366f94252907e1836ac22f4f1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/e996d940d4cd49babd94a6f624810b59] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=33.0 K 2023-04-08 02:53:47,780 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 17d3302666834341a335679dc15f7e10, keycount=100, bloomtype=ROW, size=9.3 K, encoding=NONE, compression=NONE, seqNum=9, earliestPutTs=1680922424925 2023-04-08 02:53:47,785 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 05aa5b6366f94252907e1836ac22f4f1, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=252, earliestPutTs=1680922425955 2023-04-08 02:53:47,789 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting e996d940d4cd49babd94a6f624810b59, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1680922427251 2023-04-08 02:53:47,800 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/b8a752c44d30432ba24036bdb1220c3d 2023-04-08 02:53:47,808 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#C#compaction#13 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:53:47,816 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:47,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922487815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,822 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:47,823 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39438 deadline: 1680922487819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,827 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:47,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922487826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:47,835 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:47,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39442 deadline: 1680922487832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39470 deadline: 1680922487833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,864 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=284 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/2d5ebf6211af4960a132e56f6f0c2193 2023-04-08 02:53:47,877 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/69cd0d2bf2cf46cdac670c0e2a202d09 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/69cd0d2bf2cf46cdac670c0e2a202d09 2023-04-08 02:53:47,877 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/d39bdab3821d4346b3be7e6a8517746c as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/d39bdab3821d4346b3be7e6a8517746c 2023-04-08 02:53:47,893 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/d39bdab3821d4346b3be7e6a8517746c, entries=150, sequenceid=284, filesize=11.9 K 2023-04-08 02:53:47,894 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/C of 9a2d8d09b1a5f9cc567f207bd50b4415 into 69cd0d2bf2cf46cdac670c0e2a202d09(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:53:47,899 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:47,899 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/C, priority=13, startTime=1680922427585; duration=0sec 2023-04-08 02:53:47,899 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:53:47,894 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/b8a752c44d30432ba24036bdb1220c3d as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/b8a752c44d30432ba24036bdb1220c3d 2023-04-08 02:53:47,926 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/b8a752c44d30432ba24036bdb1220c3d, entries=150, sequenceid=284, filesize=11.9 K 2023-04-08 02:53:47,930 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/2d5ebf6211af4960a132e56f6f0c2193 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/2d5ebf6211af4960a132e56f6f0c2193 2023-04-08 02:53:47,975 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/2d5ebf6211af4960a132e56f6f0c2193, entries=150, sequenceid=284, filesize=11.9 K 2023-04-08 02:53:47,981 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=80.51 KB/82440 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 309ms, sequenceid=284, compaction requested=false 2023-04-08 02:53:47,982 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:47,982 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:47,982 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:47,983 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:47,983 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,983 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:47,983 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:47,985 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:47,986 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:47,986 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:47,992 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:47,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:47,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:47,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:47,995 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55fcd3ca[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:47,995 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:47,995 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:47,999 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:47,999 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:47,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:47,999 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:47,999 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:47,999 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:47,999 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:48,002 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:48,002 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:48,002 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:48,003 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:48,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:48,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:48,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:48,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:48,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:48,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:48,010 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:48,010 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:48,010 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:48,010 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:48,010 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:48,013 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:48,013 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:48,013 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:48,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:48,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:48,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:48,014 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:48,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:48,016 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:48,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:48,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:48,025 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:48,025 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:48,025 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:48,025 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:48,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:48,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:48,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:48,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:48,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:48,029 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:48,029 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:48,030 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-08 02:53:48,025 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:48,030 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:48,030 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,030 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:53:48,030 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:48,030 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:53:48,030 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:48,030 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:53:48,030 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:48,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:48,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:48,030 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:48,032 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:48,032 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:48,065 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:48,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39442 deadline: 1680922488054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,066 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:48,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922488055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:48,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922488058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,070 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:48,070 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:48,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39470 deadline: 1680922488064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39438 deadline: 1680922488065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,097 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/3cd2840b95084349a0dc712536e2e051 2023-04-08 02:53:48,130 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:48,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:48,135 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/b6f42db44256477082734e60146a733c as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/b6f42db44256477082734e60146a733c 2023-04-08 02:53:48,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:48,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:48,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:48,148 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:48,148 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:48,149 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/A of 9a2d8d09b1a5f9cc567f207bd50b4415 into b6f42db44256477082734e60146a733c(size=12.0 K), total size for store is 23.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:53:48,149 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:48,149 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/A, priority=13, startTime=1680922427580; duration=0sec 2023-04-08 02:53:48,150 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:53:48,153 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:48,153 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:48,154 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:48,154 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:48,158 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,158 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:48,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:48,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:48,163 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,163 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:48,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:48,163 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:48,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:48,164 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:48,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:48,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39442 deadline: 1680922488168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,176 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/2e69ec9c942c41c9a7f01320bca92160 2023-04-08 02:53:48,175 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:48,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:48,177 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:48,177 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:48,175 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:48,177 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:48,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922488168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922488169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:48,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39438 deadline: 1680922488174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,178 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:48,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39470 deadline: 1680922488174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,181 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:48,182 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:48,182 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:48,182 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:48,182 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:48,182 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:48,182 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:48,182 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:48,182 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:48,182 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,182 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:48,182 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:48,184 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,185 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:48,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:48,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:48,185 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:48,185 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:48,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:48,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:48,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:48,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:48,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:48,189 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@97ca765[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:48,189 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:48,189 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:48,191 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:48,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:48,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:48,192 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:48,192 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:48,192 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:48,196 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,196 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:48,197 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,197 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:48,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:48,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:48,197 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:48,198 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:48,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:48,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:48,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:48,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:48,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:48,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:48,214 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:48,214 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:48,214 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:48,214 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:48,214 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:48,214 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:48,217 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:48,217 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:48,217 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:48,217 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:48,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:48,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:48,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:48,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:48,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:48,220 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:48,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:48,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:48,223 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:48,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,231 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:48,231 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:48,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:48,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:48,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:48,231 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:48,231 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:48,232 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,231 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:48,232 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:48,231 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:48,235 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:48,235 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:48,236 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:48,236 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:48,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:48,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:48,245 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=302 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/6ec680ae83404cb781bd8802c32badee 2023-04-08 02:53:48,259 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/3cd2840b95084349a0dc712536e2e051 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/3cd2840b95084349a0dc712536e2e051 2023-04-08 02:53:48,272 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/3cd2840b95084349a0dc712536e2e051, entries=200, sequenceid=302, filesize=14.3 K 2023-04-08 02:53:48,278 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/2e69ec9c942c41c9a7f01320bca92160 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/2e69ec9c942c41c9a7f01320bca92160 2023-04-08 02:53:48,295 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/2e69ec9c942c41c9a7f01320bca92160, entries=150, sequenceid=302, filesize=11.9 K 2023-04-08 02:53:48,298 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/6ec680ae83404cb781bd8802c32badee as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/6ec680ae83404cb781bd8802c32badee 2023-04-08 02:53:48,312 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/6ec680ae83404cb781bd8802c32badee, entries=150, sequenceid=302, filesize=11.9 K 2023-04-08 02:53:48,314 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 285ms, sequenceid=302, compaction requested=true 2023-04-08 02:53:48,314 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:48,314 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:53:48,314 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:53:48,314 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:53:48,314 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:53:48,315 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:53:48,316 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39101 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:53:48,316 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36661 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:53:48,316 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/A is initiating minor compaction (all files) 2023-04-08 02:53:48,316 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/B is initiating minor compaction (all files) 2023-04-08 02:53:48,317 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/A in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:48,317 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/B in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:48,317 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/17d35ecf0f8e4f73913d8069fb96909a, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/b8a752c44d30432ba24036bdb1220c3d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/2e69ec9c942c41c9a7f01320bca92160] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=35.8 K 2023-04-08 02:53:48,317 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/b6f42db44256477082734e60146a733c, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/d39bdab3821d4346b3be7e6a8517746c, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/3cd2840b95084349a0dc712536e2e051] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=38.2 K 2023-04-08 02:53:48,320 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 17d35ecf0f8e4f73913d8069fb96909a, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1680922427251 2023-04-08 02:53:48,321 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting b6f42db44256477082734e60146a733c, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1680922427251 2023-04-08 02:53:48,322 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting b8a752c44d30432ba24036bdb1220c3d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1680922427337 2023-04-08 02:53:48,322 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting d39bdab3821d4346b3be7e6a8517746c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1680922427337 2023-04-08 02:53:48,323 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 2e69ec9c942c41c9a7f01320bca92160, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1680922427696 2023-04-08 02:53:48,323 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 3cd2840b95084349a0dc712536e2e051, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1680922427696 2023-04-08 02:53:48,336 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:48,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:48,342 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#A#compaction#18 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:53:48,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:48,344 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#B#compaction#19 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:53:48,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:48,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:48,360 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:48,360 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:48,362 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:48,362 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:48,363 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:48,363 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:48,367 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:48,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:48,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:48,368 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:48,368 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,368 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:48,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:48,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:48,369 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:48,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:48,370 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:48,370 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:48,370 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:48,373 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:48,378 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:48,381 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:48,380 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:48,382 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:48,382 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-08 02:53:48,383 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:53:48,383 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:48,384 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:53:48,384 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:48,384 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:53:48,384 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:48,401 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:48,401 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:48,409 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/b9da0693bc53472ea93f89e46e0c77cf as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/b9da0693bc53472ea93f89e46e0c77cf 2023-04-08 02:53:48,410 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/fa682d0f3f9f4885b2d32934b5046182 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/fa682d0f3f9f4885b2d32934b5046182 2023-04-08 02:53:48,424 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/B of 9a2d8d09b1a5f9cc567f207bd50b4415 into b9da0693bc53472ea93f89e46e0c77cf(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:53:48,428 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:48,428 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/B, priority=13, startTime=1680922428314; duration=0sec 2023-04-08 02:53:48,429 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:53:48,429 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:53:48,435 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36661 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:53:48,435 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/C is initiating minor compaction (all files) 2023-04-08 02:53:48,435 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/C in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:48,435 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/69cd0d2bf2cf46cdac670c0e2a202d09, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/2d5ebf6211af4960a132e56f6f0c2193, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/6ec680ae83404cb781bd8802c32badee] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=35.8 K 2023-04-08 02:53:48,438 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 69cd0d2bf2cf46cdac670c0e2a202d09, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=265, earliestPutTs=1680922427251 2023-04-08 02:53:48,439 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 2d5ebf6211af4960a132e56f6f0c2193, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=284, earliestPutTs=1680922427337 2023-04-08 02:53:48,440 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/A of 9a2d8d09b1a5f9cc567f207bd50b4415 into fa682d0f3f9f4885b2d32934b5046182(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:53:48,440 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:48,440 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/A, priority=13, startTime=1680922428314; duration=0sec 2023-04-08 02:53:48,440 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:53:48,441 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 6ec680ae83404cb781bd8802c32badee, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1680922427696 2023-04-08 02:53:48,449 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:48,449 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:48,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39442 deadline: 1680922488430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922488430, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,450 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:48,450 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:48,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39470 deadline: 1680922488435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922488439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,451 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:48,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39438 deadline: 1680922488441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,471 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#C#compaction#21 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:53:48,506 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/fbef1f516d8844b49999a61d16459d8f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/fbef1f516d8844b49999a61d16459d8f 2023-04-08 02:53:48,517 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/C of 9a2d8d09b1a5f9cc567f207bd50b4415 into fbef1f516d8844b49999a61d16459d8f(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:53:48,517 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:48,517 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/C, priority=13, startTime=1680922428315; duration=0sec 2023-04-08 02:53:48,520 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:53:48,554 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:48,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39442 deadline: 1680922488552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,555 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:48,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922488552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:48,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922488553, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,556 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:48,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39470 deadline: 1680922488554, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:48,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39438 deadline: 1680922488558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,760 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:48,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:48,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39442 deadline: 1680922488758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922488758, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,761 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:48,761 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:48,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922488759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39470 deadline: 1680922488759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,766 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:48,766 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39438 deadline: 1680922488764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:48,827 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/aaeb958413ee4a7eb33a80acf67feb63 2023-04-08 02:53:48,864 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/ef8fb2c75d714df19926c27c8e058aa5 2023-04-08 02:53:49,066 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:49,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922489063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,067 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:49,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:49,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39442 deadline: 1680922489064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922489064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,068 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:49,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39470 deadline: 1680922489066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,076 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:49,076 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39438 deadline: 1680922489071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,312 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=320 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/2752988baa374592a6d98020fd24f2db 2023-04-08 02:53:49,323 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/aaeb958413ee4a7eb33a80acf67feb63 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/aaeb958413ee4a7eb33a80acf67feb63 2023-04-08 02:53:49,332 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/aaeb958413ee4a7eb33a80acf67feb63, entries=150, sequenceid=320, filesize=11.9 K 2023-04-08 02:53:49,335 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/ef8fb2c75d714df19926c27c8e058aa5 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/ef8fb2c75d714df19926c27c8e058aa5 2023-04-08 02:53:49,346 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/ef8fb2c75d714df19926c27c8e058aa5, entries=150, sequenceid=320, filesize=11.9 K 2023-04-08 02:53:49,347 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/2752988baa374592a6d98020fd24f2db as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/2752988baa374592a6d98020fd24f2db 2023-04-08 02:53:49,368 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/2752988baa374592a6d98020fd24f2db, entries=150, sequenceid=320, filesize=11.9 K 2023-04-08 02:53:49,376 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 993ms, sequenceid=320, compaction requested=false 2023-04-08 02:53:49,376 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:49,376 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:49,376 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:49,378 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:49,378 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,378 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:49,378 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:49,385 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:49,385 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:49,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:49,386 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:49,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:49,386 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:49,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:49,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:49,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:49,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:49,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ee21ab2[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:49,389 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:49,389 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:49,391 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:49,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:49,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:49,391 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:49,391 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:49,391 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:49,393 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,393 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:49,394 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:49,395 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,395 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:49,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:49,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:49,395 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:49,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:49,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:49,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:49,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:49,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:49,401 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:49,401 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:49,401 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:49,401 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:49,401 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:49,406 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:49,406 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:49,406 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:49,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:49,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:49,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:49,408 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:49,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:49,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:49,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:49,411 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:49,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:49,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,418 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:49,418 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:49,418 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:49,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:49,419 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,419 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:49,419 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:49,419 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,419 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:49,419 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:49,419 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:49,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:49,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:49,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:49,422 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:49,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:49,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:49,424 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:49,424 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:49,524 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:49,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:49,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:49,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:49,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:49,546 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:49,546 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:49,547 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:49,547 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:49,548 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:49,548 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:49,550 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:49,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:49,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:49,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:49,551 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,551 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:49,551 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,551 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:49,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:49,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:49,552 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:49,558 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:49,558 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:49,558 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:49,559 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:49,560 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:49,572 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:49,572 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:49,572 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:49,572 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:49,572 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-08 02:53:49,573 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:53:49,573 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:49,573 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:53:49,573 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:49,573 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:53:49,573 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:49,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:49,577 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:49,611 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:49,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922489596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,612 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:49,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39442 deadline: 1680922489597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:49,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922489601, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,613 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:49,614 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39470 deadline: 1680922489611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,617 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:49,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39438 deadline: 1680922489616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,715 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:49,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922489714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,716 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:49,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39442 deadline: 1680922489714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:49,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922489715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:49,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39470 deadline: 1680922489715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:49,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39438 deadline: 1680922489719, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,920 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:49,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39442 deadline: 1680922489918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,921 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:49,921 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922489919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,924 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:49,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39470 deadline: 1680922489921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,925 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:49,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922489921, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:49,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:49,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39438 deadline: 1680922489926, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,013 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/dd588e641ae5448b876e467fa8703acd 2023-04-08 02:53:50,051 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/6fb1dce0e3a14cc88e93d192f0baf7cb 2023-04-08 02:53:50,079 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=338 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/96aa0093de3242d5a26a998cef5faab7 2023-04-08 02:53:50,091 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/dd588e641ae5448b876e467fa8703acd as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/dd588e641ae5448b876e467fa8703acd 2023-04-08 02:53:50,100 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/dd588e641ae5448b876e467fa8703acd, entries=150, sequenceid=338, filesize=11.9 K 2023-04-08 02:53:50,108 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/6fb1dce0e3a14cc88e93d192f0baf7cb as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/6fb1dce0e3a14cc88e93d192f0baf7cb 2023-04-08 02:53:50,115 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/6fb1dce0e3a14cc88e93d192f0baf7cb, entries=150, sequenceid=338, filesize=11.9 K 2023-04-08 02:53:50,119 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/96aa0093de3242d5a26a998cef5faab7 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/96aa0093de3242d5a26a998cef5faab7 2023-04-08 02:53:50,128 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/96aa0093de3242d5a26a998cef5faab7, entries=150, sequenceid=338, filesize=11.9 K 2023-04-08 02:53:50,129 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 557ms, sequenceid=338, compaction requested=true 2023-04-08 02:53:50,130 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:50,130 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:50,130 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:50,130 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:50,130 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,131 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:50,131 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:50,132 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,132 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:50,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:50,133 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,133 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:50,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:50,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:50,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:50,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:50,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@756fea8a[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:50,137 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:50,137 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,138 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,138 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:50,138 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:50,138 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:50,138 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,138 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:50,140 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:50,140 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,141 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:50,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:50,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:50,141 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:50,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:50,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:50,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:50,145 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:50,145 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:50,145 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:50,145 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:50,145 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:50,145 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:50,148 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,148 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:50,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:50,148 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:50,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:50,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:50,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,152 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,157 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,160 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,160 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:50,160 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:50,160 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:50,160 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:50,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:50,160 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:50,161 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,161 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,161 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:50,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:50,168 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:50,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:50,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:50,173 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:50,173 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:50,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:50,228 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-08 02:53:50,230 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:53:50,230 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:50,230 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:53:50,230 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:50,230 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:53:50,230 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:50,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:50,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39442 deadline: 1680922490241, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:50,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922490244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,254 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:50,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922490244, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:50,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39470 deadline: 1680922490245, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,257 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/72319d67dc5e40a0b8e2109413912deb 2023-04-08 02:53:50,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:50,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39438 deadline: 1680922490254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,273 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:50,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:50,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:50,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:50,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:50,342 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:50,342 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:50,349 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:50,349 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:50,349 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,350 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:50,351 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,351 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:50,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:50,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:50,351 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,351 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:50,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:50,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:50,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:50,353 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:50,361 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:50,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39442 deadline: 1680922490356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,362 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:50,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922490356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,362 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:50,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922490357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,363 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:50,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39470 deadline: 1680922490357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,366 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:50,366 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:50,366 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:50,366 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:50,366 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:50,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39438 deadline: 1680922490362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,369 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:50,388 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:50,395 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:50,395 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:50,404 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:50,404 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:50,404 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:50,404 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:50,404 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:50,404 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,404 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:50,404 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:50,408 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,408 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:50,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:50,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:50,410 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,410 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:50,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:50,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:50,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:50,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@463e156e[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:50,414 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:50,414 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,416 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,416 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:50,416 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:50,416 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:50,417 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,417 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:50,418 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:50,418 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:50,418 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:50,418 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:50,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:50,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:50,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:50,435 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:50,435 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:50,435 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:50,435 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:50,435 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:50,437 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,437 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,437 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:50,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:50,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:50,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:50,438 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:50,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:50,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,441 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:50,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,451 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:50,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:50,451 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:50,452 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,452 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:50,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:50,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:50,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:50,452 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:50,453 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:50,453 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:50,454 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:50,452 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:50,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:50,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:50,453 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,454 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,454 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,454 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,554 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:50,555 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:50,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:50,566 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:50,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922490564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:50,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39442 deadline: 1680922490564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:50,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922490564, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,568 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:50,568 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39470 deadline: 1680922490565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,578 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:50,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39438 deadline: 1680922490573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:50,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:50,579 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:50,579 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:50,581 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:50,581 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:50,582 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,582 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:50,584 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:50,584 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:50,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:50,584 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,585 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:50,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:50,585 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:50,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:50,586 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:50,599 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:50,599 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:50,599 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:50,599 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:50,605 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:50,605 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:50,605 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:50,605 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:50,605 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:50,606 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:50,606 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:50,606 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:50,606 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:50,606 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,606 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:50,606 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:50,608 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,608 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:50,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:50,609 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,609 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:50,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:50,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:50,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:50,614 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a15c543[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:50,614 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:50,614 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,616 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,616 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:50,616 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:50,616 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:50,617 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,617 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:50,619 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,619 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:50,620 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,620 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:50,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:50,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:50,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:50,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:50,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:50,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:50,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:50,630 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:50,630 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:50,630 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:50,630 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:50,630 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:50,633 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,633 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:50,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,634 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:50,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:50,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:50,635 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:50,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,640 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,644 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:50,644 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:50,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:50,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:50,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:50,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:50,644 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:50,649 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:50,649 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:50,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:50,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:50,652 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,652 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:50,652 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:50,652 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,652 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,652 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,652 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,750 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:50,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:50,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:50,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:50,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:50,756 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:50,756 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:50,757 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:50,757 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:50,758 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,758 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:50,759 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:50,759 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:50,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:50,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:50,760 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,760 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:50,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:50,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:50,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:50,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:50,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:50,764 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:50,764 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:50,767 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:50,767 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:50,767 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:50,767 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:50,767 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:50,768 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:50,768 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:50,768 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:50,768 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:50,768 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,768 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:50,768 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:50,770 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,770 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:50,771 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,771 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:50,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:50,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:50,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@44cc6d39[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:50,773 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:50,773 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,775 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,775 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,775 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:50,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,775 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:50,775 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:50,775 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:50,778 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:50,778 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:50,778 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:50,782 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:50,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:50,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:50,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:50,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:50,787 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:50,787 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:50,787 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:50,787 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:50,787 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:50,787 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:50,789 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,789 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:50,789 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:50,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:50,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:50,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:50,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:50,790 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:50,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,794 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:50,795 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/bd358646e2a444fe92a07cc014e61202 2023-04-08 02:53:50,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,807 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:50,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:50,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:50,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:50,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:50,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:50,808 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:50,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:50,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:50,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:50,809 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:50,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:50,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:50,809 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:50,836 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=358 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/b9866a1a04764d72974b273dd733a90b 2023-04-08 02:53:50,848 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/72319d67dc5e40a0b8e2109413912deb as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/72319d67dc5e40a0b8e2109413912deb 2023-04-08 02:53:50,857 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/72319d67dc5e40a0b8e2109413912deb, entries=150, sequenceid=358, filesize=11.9 K 2023-04-08 02:53:50,859 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/bd358646e2a444fe92a07cc014e61202 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/bd358646e2a444fe92a07cc014e61202 2023-04-08 02:53:50,867 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/bd358646e2a444fe92a07cc014e61202, entries=150, sequenceid=358, filesize=11.9 K 2023-04-08 02:53:50,871 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/b9866a1a04764d72974b273dd733a90b as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/b9866a1a04764d72974b273dd733a90b 2023-04-08 02:53:50,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:50,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922490868, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:50,873 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39442 deadline: 1680922490870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,873 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:50,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922490871, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,874 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:50,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39470 deadline: 1680922490872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,879 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/b9866a1a04764d72974b273dd733a90b, entries=150, sequenceid=358, filesize=11.9 K 2023-04-08 02:53:50,881 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 653ms, sequenceid=358, compaction requested=true 2023-04-08 02:53:50,881 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:50,881 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:53:50,881 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:53:50,881 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:53:50,881 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-08 02:53:50,881 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:53:50,883 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48950 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:53:50,883 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48950 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:53:50,883 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/B is initiating minor compaction (all files) 2023-04-08 02:53:50,883 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/A is initiating minor compaction (all files) 2023-04-08 02:53:50,883 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/B in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:50,883 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/A in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:50,884 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/b9da0693bc53472ea93f89e46e0c77cf, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/ef8fb2c75d714df19926c27c8e058aa5, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/6fb1dce0e3a14cc88e93d192f0baf7cb, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/bd358646e2a444fe92a07cc014e61202] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=47.8 K 2023-04-08 02:53:50,884 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/fa682d0f3f9f4885b2d32934b5046182, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/aaeb958413ee4a7eb33a80acf67feb63, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/dd588e641ae5448b876e467fa8703acd, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/72319d67dc5e40a0b8e2109413912deb] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=47.8 K 2023-04-08 02:53:50,885 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting fa682d0f3f9f4885b2d32934b5046182, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1680922427696 2023-04-08 02:53:50,886 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting aaeb958413ee4a7eb33a80acf67feb63, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680922428051 2023-04-08 02:53:50,886 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting dd588e641ae5448b876e467fa8703acd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1680922428429 2023-04-08 02:53:50,886 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting b9da0693bc53472ea93f89e46e0c77cf, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1680922427696 2023-04-08 02:53:50,888 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 72319d67dc5e40a0b8e2109413912deb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1680922429597 2023-04-08 02:53:50,888 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting ef8fb2c75d714df19926c27c8e058aa5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680922428051 2023-04-08 02:53:50,889 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 6fb1dce0e3a14cc88e93d192f0baf7cb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1680922428429 2023-04-08 02:53:50,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:50,889 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting bd358646e2a444fe92a07cc014e61202, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1680922429597 2023-04-08 02:53:50,890 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-08 02:53:50,891 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:53:50,891 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:50,891 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:53:50,891 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:50,891 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:53:50,891 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:50,909 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:50,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:50,918 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/c025dc22230c4ddda694ca4e39267b4f 2023-04-08 02:53:50,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:50,920 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#B#compaction#32 average throughput is 0.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:53:50,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:50,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:50,923 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:50,923 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:50,924 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:50,924 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:50,924 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,924 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:50,925 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#A#compaction#31 average throughput is 0.50 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:53:50,926 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:50,926 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:50,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:50,927 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,927 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:50,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:50,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:50,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:50,928 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:50,933 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:50,933 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:50,933 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:50,933 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:50,934 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:50,935 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:50,935 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:50,935 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:50,940 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:50,940 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:50,940 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:50,941 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:50,941 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:50,941 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,948 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:50,948 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:50,950 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,950 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:50,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:50,951 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,951 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:50,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:50,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:50,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:50,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e7de490[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:50,954 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:50,954 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,956 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,956 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:50,956 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:50,956 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:50,956 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,956 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:50,959 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,959 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:50,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,959 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:50,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:50,959 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:50,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:50,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:50,968 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:50,968 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:50,968 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:50,968 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:50,968 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:50,968 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:50,970 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,970 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:50,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,970 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:50,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:50,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:50,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:50,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:50,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,971 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:50,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,976 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/ac381fac7b9549348a0ad441bc65cd43 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/ac381fac7b9549348a0ad441bc65cd43 2023-04-08 02:53:50,978 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:50,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:50,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,984 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:50,984 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:50,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:50,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:50,984 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:50,984 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:50,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:50,984 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:50,984 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:50,984 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:50,984 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:50,984 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:50,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:50,984 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:50,985 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:50,985 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:50,992 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/A of 9a2d8d09b1a5f9cc567f207bd50b4415 into ac381fac7b9549348a0ad441bc65cd43(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:53:50,993 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:50,993 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/A, priority=12, startTime=1680922430881; duration=0sec 2023-04-08 02:53:50,993 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:53:50,993 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:53:50,998 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/93be6ee218604988bb2f770e1d61860f 2023-04-08 02:53:50,998 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48950 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:53:50,998 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/C is initiating minor compaction (all files) 2023-04-08 02:53:50,998 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/C in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:50,998 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/fbef1f516d8844b49999a61d16459d8f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/2752988baa374592a6d98020fd24f2db, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/96aa0093de3242d5a26a998cef5faab7, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/b9866a1a04764d72974b273dd733a90b] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=47.8 K 2023-04-08 02:53:50,999 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting fbef1f516d8844b49999a61d16459d8f, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=302, earliestPutTs=1680922427696 2023-04-08 02:53:51,000 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 2752988baa374592a6d98020fd24f2db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=320, earliestPutTs=1680922428051 2023-04-08 02:53:51,000 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 96aa0093de3242d5a26a998cef5faab7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=338, earliestPutTs=1680922428429 2023-04-08 02:53:51,001 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting b9866a1a04764d72974b273dd733a90b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1680922429597 2023-04-08 02:53:51,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:51,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922490999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,026 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#C#compaction#35 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:53:51,034 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=371 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/065d01042c614202843c740fdb4c1b71 2023-04-08 02:53:51,051 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/c025dc22230c4ddda694ca4e39267b4f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/c025dc22230c4ddda694ca4e39267b4f 2023-04-08 02:53:51,057 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/f07537a5786c46bfb0b8a20caddb61e3 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/f07537a5786c46bfb0b8a20caddb61e3 2023-04-08 02:53:51,062 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/c025dc22230c4ddda694ca4e39267b4f, entries=150, sequenceid=371, filesize=11.9 K 2023-04-08 02:53:51,065 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/93be6ee218604988bb2f770e1d61860f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/93be6ee218604988bb2f770e1d61860f 2023-04-08 02:53:51,077 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/C of 9a2d8d09b1a5f9cc567f207bd50b4415 into f07537a5786c46bfb0b8a20caddb61e3(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:53:51,077 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:51,078 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/C, priority=12, startTime=1680922430881; duration=0sec 2023-04-08 02:53:51,078 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:53:51,081 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/93be6ee218604988bb2f770e1d61860f, entries=150, sequenceid=371, filesize=11.9 K 2023-04-08 02:53:51,085 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:51,085 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/065d01042c614202843c740fdb4c1b71 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/065d01042c614202843c740fdb4c1b71 2023-04-08 02:53:51,086 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:51,089 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:51,102 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/065d01042c614202843c740fdb4c1b71, entries=150, sequenceid=371, filesize=11.9 K 2023-04-08 02:53:51,103 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 213ms, sequenceid=371, compaction requested=false 2023-04-08 02:53:51,103 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:51,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:51,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:51,108 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:51,108 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:51,109 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:51,109 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:51,114 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,114 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:51,116 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:51,116 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,116 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:51,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:51,117 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,117 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:51,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:51,117 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:51,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:51,118 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:51,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:51,119 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-08 02:53:51,119 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:53:51,119 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:51,119 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:53:51,119 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:51,119 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:53:51,119 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:51,122 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:51,122 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:51,122 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:51,123 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:51,123 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:51,124 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:51,124 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:51,124 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:51,124 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:51,124 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:51,124 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:51,125 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:51,125 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:51,125 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,125 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:51,125 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:51,130 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:51,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:51,131 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:51,131 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,131 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:51,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:51,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:51,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78de0d54[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:51,133 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:51,133 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,135 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,135 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:51,135 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:51,135 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:51,136 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,136 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:51,137 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:51,137 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:51,137 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:51,142 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:51,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:51,144 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/45dbb59b772d49cca43b62e8a2d201c7 2023-04-08 02:53:51,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:51,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:51,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:51,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:51,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:51,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:51,154 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:51,154 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:51,154 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:51,154 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:51,154 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:51,157 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,157 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:51,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,158 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:51,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:51,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:51,159 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:51,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:51,161 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:51,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:51,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,167 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/c9f2fa8a61b1418b9b59759c0901caf5 2023-04-08 02:53:51,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:51,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:51,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:51,169 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:51,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:51,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:51,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:51,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:51,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:51,171 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,171 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,171 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,171 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,171 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:51,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:51,171 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:51,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:51,185 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:51,185 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922491184, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,231 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=393 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/7922ae0f305f4d99aeca9b72beb93be4 2023-04-08 02:53:51,239 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/45dbb59b772d49cca43b62e8a2d201c7 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/45dbb59b772d49cca43b62e8a2d201c7 2023-04-08 02:53:51,246 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/45dbb59b772d49cca43b62e8a2d201c7, entries=200, sequenceid=393, filesize=14.3 K 2023-04-08 02:53:51,247 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/c9f2fa8a61b1418b9b59759c0901caf5 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/c9f2fa8a61b1418b9b59759c0901caf5 2023-04-08 02:53:51,256 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/c9f2fa8a61b1418b9b59759c0901caf5, entries=150, sequenceid=393, filesize=11.9 K 2023-04-08 02:53:51,257 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/7922ae0f305f4d99aeca9b72beb93be4 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/7922ae0f305f4d99aeca9b72beb93be4 2023-04-08 02:53:51,268 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/7922ae0f305f4d99aeca9b72beb93be4, entries=150, sequenceid=393, filesize=11.9 K 2023-04-08 02:53:51,270 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 151ms, sequenceid=393, compaction requested=true 2023-04-08 02:53:51,270 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:51,270 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:53:51,270 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:53:51,270 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:53:51,270 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-08 02:53:51,271 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:51,272 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39340 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:53:51,272 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/A is initiating minor compaction (all files) 2023-04-08 02:53:51,272 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/A in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:51,272 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/ac381fac7b9549348a0ad441bc65cd43, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/c025dc22230c4ddda694ca4e39267b4f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/45dbb59b772d49cca43b62e8a2d201c7] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=38.4 K 2023-04-08 02:53:51,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:51,272 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting ac381fac7b9549348a0ad441bc65cd43, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1680922429597 2023-04-08 02:53:51,274 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting c025dc22230c4ddda694ca4e39267b4f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1680922430232 2023-04-08 02:53:51,275 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 45dbb59b772d49cca43b62e8a2d201c7, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1680922430982 2023-04-08 02:53:51,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:51,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:51,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:51,279 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:51,279 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:51,281 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:51,281 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:51,281 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,281 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:51,283 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:51,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:51,283 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:51,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:51,284 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,284 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:51,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:51,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:51,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:51,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:51,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:51,285 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:51,285 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:51,286 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:51,286 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:51,286 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:51,287 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:51,287 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:51,287 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-08 02:53:51,288 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:53:51,288 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:51,288 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:53:51,288 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:51,288 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:53:51,288 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:51,289 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#A#compaction#39 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:53:51,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:51,296 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:51,310 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/123a121f9639404f81c7b5dcad954138 2023-04-08 02:53:51,342 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/1b34b86b131f40018e17518fc1c422db 2023-04-08 02:53:51,370 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/b14d090365884d4a9c24e4d282935e9b as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/b14d090365884d4a9c24e4d282935e9b 2023-04-08 02:53:51,378 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/B of 9a2d8d09b1a5f9cc567f207bd50b4415 into b14d090365884d4a9c24e4d282935e9b(size=12.2 K), total size for store is 36.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:53:51,378 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:51,378 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/B, priority=12, startTime=1680922430881; duration=0sec 2023-04-08 02:53:51,379 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-08 02:53:51,379 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:53:51,380 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36900 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:53:51,381 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/C is initiating minor compaction (all files) 2023-04-08 02:53:51,381 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/C in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:51,381 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/f07537a5786c46bfb0b8a20caddb61e3, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/065d01042c614202843c740fdb4c1b71, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/7922ae0f305f4d99aeca9b72beb93be4] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=36.0 K 2023-04-08 02:53:51,381 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting f07537a5786c46bfb0b8a20caddb61e3, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1680922429597 2023-04-08 02:53:51,382 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 065d01042c614202843c740fdb4c1b71, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1680922430232 2023-04-08 02:53:51,383 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 7922ae0f305f4d99aeca9b72beb93be4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1680922430993 2023-04-08 02:53:51,394 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680922431357, max timestamp=1680922431385Num uniques 859; 2023-04-08 02:53:51,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:51,399 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:51,396 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:51,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680922431357, max timestamp=1680922431385Num uniques 859; 2023-04-08 02:53:51,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39470 deadline: 1680922491378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-08 02:53:51,395 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680922431357, max timestamp=1680922431385Num uniques 859; 2023-04-08 02:53:51,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922491379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-08 02:53:51,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922491386, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,405 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-08 02:53:51,407 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=405 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/df79a0352066452a9ad7487c71d92604 2023-04-08 02:53:51,409 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#C#compaction#43 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:53:51,420 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/123a121f9639404f81c7b5dcad954138 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/123a121f9639404f81c7b5dcad954138 2023-04-08 02:53:51,429 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/123a121f9639404f81c7b5dcad954138, entries=150, sequenceid=405, filesize=11.9 K 2023-04-08 02:53:51,431 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/1b34b86b131f40018e17518fc1c422db as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/1b34b86b131f40018e17518fc1c422db 2023-04-08 02:53:51,450 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/1b34b86b131f40018e17518fc1c422db, entries=150, sequenceid=405, filesize=11.9 K 2023-04-08 02:53:51,451 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/df79a0352066452a9ad7487c71d92604 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/df79a0352066452a9ad7487c71d92604 2023-04-08 02:53:51,466 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/49b116944fcf4d17a7b0275b236bab03 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/49b116944fcf4d17a7b0275b236bab03 2023-04-08 02:53:51,479 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/df79a0352066452a9ad7487c71d92604, entries=150, sequenceid=405, filesize=11.9 K 2023-04-08 02:53:51,494 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=93.93 KB/96180 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 207ms, sequenceid=405, compaction requested=true 2023-04-08 02:53:51,494 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:51,494 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:51,494 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:51,494 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:51,494 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,494 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:51,495 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:51,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:51,495 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=93.93 KB heapSize=256.08 KB 2023-04-08 02:53:51,496 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:53:51,496 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:53:51,496 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:53:51,496 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:53:51,496 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:53:51,496 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:53:51,496 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:51,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:51,497 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:51,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:51,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:51,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:51,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ce5f0d1[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:51,500 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:51,501 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,502 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,502 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:51,502 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:51,502 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:51,503 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,503 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:51,505 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,505 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:51,505 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:51,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:51,505 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:51,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:51,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:51,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:51,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,514 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/C of 9a2d8d09b1a5f9cc567f207bd50b4415 into 49b116944fcf4d17a7b0275b236bab03(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:53:51,514 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:51,514 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/C, priority=13, startTime=1680922431270; duration=0sec 2023-04-08 02:53:51,514 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:53:51,514 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:53:51,518 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49086 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:53:51,518 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/B is initiating minor compaction (all files) 2023-04-08 02:53:51,518 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/B in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:51,518 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/b14d090365884d4a9c24e4d282935e9b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/93be6ee218604988bb2f770e1d61860f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/c9f2fa8a61b1418b9b59759c0901caf5, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/1b34b86b131f40018e17518fc1c422db] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=47.9 K 2023-04-08 02:53:51,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:51,519 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting b14d090365884d4a9c24e4d282935e9b, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=358, earliestPutTs=1680922429597 2023-04-08 02:53:51,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,542 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 93be6ee218604988bb2f770e1d61860f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=371, earliestPutTs=1680922430232 2023-04-08 02:53:51,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:51,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:51,543 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:51,543 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:51,543 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:51,543 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:51,543 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:51,543 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting c9f2fa8a61b1418b9b59759c0901caf5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1680922430993 2023-04-08 02:53:51,544 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 1b34b86b131f40018e17518fc1c422db, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1680922431144 2023-04-08 02:53:51,545 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,545 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:51,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,546 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:51,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:51,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:51,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:51,548 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:51,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:51,550 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:51,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:51,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,554 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,554 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:51,554 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:51,554 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:51,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:51,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:51,555 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,555 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,555 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,555 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:51,555 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:51,555 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:51,559 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:51,559 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:51,559 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:51,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:51,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:51,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:51,568 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=435 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/9fd2e1f5a9944ad5851b16bfbf0d2c35 2023-04-08 02:53:51,578 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#B#compaction#45 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:53:51,581 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:51,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922491569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,582 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:51,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922491569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,587 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:51,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922491577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,593 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:51,593 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922491588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,604 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/c3858b3e688e45e483b5b0de8835a937 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/c3858b3e688e45e483b5b0de8835a937 2023-04-08 02:53:51,611 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/B of 9a2d8d09b1a5f9cc567f207bd50b4415 into c3858b3e688e45e483b5b0de8835a937(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:53:51,611 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:51,612 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/B, priority=12, startTime=1680922431270; duration=0sec 2023-04-08 02:53:51,612 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:53:51,655 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:51,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:51,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:51,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:51,667 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:51,667 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:51,667 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:51,669 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:51,669 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:51,670 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,670 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:51,673 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,673 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,673 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:51,675 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:51,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:51,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:51,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:51,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:51,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:51,677 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:51,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:51,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:51,680 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:51,680 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:51,685 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:51,685 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:51,685 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:51,685 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:51,685 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:51,685 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:51,685 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:51,685 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:51,685 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:51,685 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,685 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:51,686 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:51,688 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,689 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,689 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:51,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:51,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922491686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:51,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922491686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,691 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:51,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922491689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,693 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:51,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:51,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:51,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:51,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:51,697 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b67efd1[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:51,697 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:51,697 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,698 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,698 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:51,698 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:51,698 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:51,699 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,699 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:51,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:51,700 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922491695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:51,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:51,700 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:51,700 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:51,700 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:51,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:51,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:51,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:51,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:51,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:51,705 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:51,705 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:51,705 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:51,705 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:51,705 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:51,711 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,711 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:51,711 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,711 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:51,712 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:51,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,713 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:51,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:51,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:51,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,719 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/5faee44d217b4f0b83baf62a4713bc91 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/5faee44d217b4f0b83baf62a4713bc91 2023-04-08 02:53:51,720 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:51,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:51,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,724 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,724 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:51,724 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:51,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:51,724 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:51,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:51,724 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:51,724 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:51,724 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:51,724 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,724 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,724 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:51,724 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,724 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:51,729 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:51,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:51,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:51,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:51,736 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/A of 9a2d8d09b1a5f9cc567f207bd50b4415 into 5faee44d217b4f0b83baf62a4713bc91(size=12.3 K), total size for store is 24.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:53:51,736 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:51,736 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/A, priority=13, startTime=1680922431270; duration=0sec 2023-04-08 02:53:51,737 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:53:51,829 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:51,832 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:51,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:51,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:51,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:51,844 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:51,844 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:51,849 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:51,849 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:51,849 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,850 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:51,852 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,853 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:51,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:51,853 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,853 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:51,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:51,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:51,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:51,855 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:51,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:51,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:51,855 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:51,855 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:51,857 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:51,860 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:51,860 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:51,860 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:51,860 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:51,860 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:51,860 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:51,860 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:51,860 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:51,860 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,860 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:51,861 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:51,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,862 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:51,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,863 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:51,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:51,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:51,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:51,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:51,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:51,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a1fc949[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:51,868 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:51,869 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,877 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,878 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:51,878 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:51,878 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:51,878 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,878 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:51,880 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:51,880 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:51,880 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:51,881 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:51,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:51,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:51,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:51,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:51,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:51,894 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:51,894 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:51,894 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:51,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922491891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,895 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:51,895 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:51,896 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:51,896 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:51,896 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:51,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922491891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:51,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922491892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,900 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,900 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:51,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,900 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:51,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:51,901 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:51,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:51,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:51,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:51,903 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:51,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:51,906 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:51,906 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39458 deadline: 1680922491903, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,907 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:51,907 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:51,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:51,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:51,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:51,908 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,908 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:51,908 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:51,908 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:51,908 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,908 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:51,908 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:51,908 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:51,908 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:51,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:51,922 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:51,922 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:51,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:52,021 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=435 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/ffea607b14e74d33838cd3974cb74f21 2023-04-08 02:53:52,022 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:52,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:52,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:52,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:52,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:52,040 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:52,040 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:52,043 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:52,043 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:52,044 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,044 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,046 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:52,046 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:52,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:52,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,048 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:52,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,049 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:52,049 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,049 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:52,052 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:52,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:52,052 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:52,053 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:52,062 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:52,062 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:52,063 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:52,063 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:52,063 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:52,063 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:52,064 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:52,064 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:52,064 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:52,064 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,064 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:52,064 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:52,069 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:52,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:52,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,071 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:52,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:52,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72a3cff3[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:52,072 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:52,072 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,072 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,072 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:52,073 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,073 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,074 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:52,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,076 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:52,076 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:52,076 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:52,082 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:52,082 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:52,082 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:52,082 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:52,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:52,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:52,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,097 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=435 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/55fc33fdaf6a4e3c9cc7a6fff56bfd5b 2023-04-08 02:53:52,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:52,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:52,106 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:52,106 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:52,106 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:52,106 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:52,106 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:52,112 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,112 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:52,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,112 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:52,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:52,113 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/9fd2e1f5a9944ad5851b16bfbf0d2c35 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/9fd2e1f5a9944ad5851b16bfbf0d2c35 2023-04-08 02:53:52,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:52,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:52,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:52,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,117 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:52,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:52,119 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,124 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,124 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,124 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,124 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,124 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,124 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,124 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,124 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:52,124 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:52,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:52,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:52,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:52,124 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:52,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:52,124 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:52,126 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:52,126 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:52,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:52,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:52,127 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/9fd2e1f5a9944ad5851b16bfbf0d2c35, entries=150, sequenceid=435, filesize=11.9 K 2023-04-08 02:53:52,130 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/ffea607b14e74d33838cd3974cb74f21 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/ffea607b14e74d33838cd3974cb74f21 2023-04-08 02:53:52,140 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/ffea607b14e74d33838cd3974cb74f21, entries=150, sequenceid=435, filesize=11.9 K 2023-04-08 02:53:52,142 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/55fc33fdaf6a4e3c9cc7a6fff56bfd5b as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/55fc33fdaf6a4e3c9cc7a6fff56bfd5b 2023-04-08 02:53:52,152 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/55fc33fdaf6a4e3c9cc7a6fff56bfd5b, entries=150, sequenceid=435, filesize=11.9 K 2023-04-08 02:53:52,153 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~256.03 KB/262176, currentSize=93.93 KB/96180 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 658ms, sequenceid=435, compaction requested=true 2023-04-08 02:53:52,153 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:52,153 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:53:52,154 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:53:52,154 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-08 02:53:52,154 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:53:52,155 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:53:52,155 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/A is initiating minor compaction (all files) 2023-04-08 02:53:52,155 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-08 02:53:52,155 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:53:52,155 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-08 02:53:52,156 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. because compaction request was cancelled 2023-04-08 02:53:52,156 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:53:52,155 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/A in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:52,156 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/5faee44d217b4f0b83baf62a4713bc91, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/123a121f9639404f81c7b5dcad954138, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/9fd2e1f5a9944ad5851b16bfbf0d2c35] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=36.1 K 2023-04-08 02:53:52,158 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 5faee44d217b4f0b83baf62a4713bc91, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1680922430993 2023-04-08 02:53:52,159 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 123a121f9639404f81c7b5dcad954138, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1680922431144 2023-04-08 02:53:52,160 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37002 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:53:52,171 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/C is initiating minor compaction (all files) 2023-04-08 02:53:52,171 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 9fd2e1f5a9944ad5851b16bfbf0d2c35, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1680922431457 2023-04-08 02:53:52,172 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/C in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:52,172 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/49b116944fcf4d17a7b0275b236bab03, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/df79a0352066452a9ad7487c71d92604, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/55fc33fdaf6a4e3c9cc7a6fff56bfd5b] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=36.1 K 2023-04-08 02:53:52,173 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 49b116944fcf4d17a7b0275b236bab03, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=393, earliestPutTs=1680922430993 2023-04-08 02:53:52,174 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting df79a0352066452a9ad7487c71d92604, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1680922431144 2023-04-08 02:53:52,175 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 55fc33fdaf6a4e3c9cc7a6fff56bfd5b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1680922431457 2023-04-08 02:53:52,187 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#A#compaction#48 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:53:52,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:52,200 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-08 02:53:52,200 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#C#compaction#49 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:53:52,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:53:52,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:52,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:53:52,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:52,201 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:53:52,201 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:52,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:52,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922492217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:52,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922492217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,223 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:52,224 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922492218, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,225 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:52,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922492219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,226 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:52,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:52,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:52,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:52,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:52,256 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:52,256 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:52,259 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:52,259 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:52,260 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,260 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,261 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:52,261 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:52,261 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,261 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:52,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:52,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,262 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:52,262 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:52,263 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:52,263 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:52,263 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:52,264 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:52,264 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:52,264 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:52,264 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:52,264 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:52,264 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:52,264 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:52,264 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:52,264 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:52,264 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,264 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:52,264 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:52,266 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,266 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:52,266 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,266 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:52,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:52,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:52,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2113ca0e[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:52,268 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:52,268 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,269 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,269 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:52,269 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,269 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:52,269 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:52,270 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:52,272 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,272 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:52,272 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:52,272 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:52,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:52,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:52,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:52,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:52,278 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:52,278 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:52,278 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:52,278 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:52,278 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:52,278 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:52,280 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,280 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,280 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:52,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:52,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:52,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:52,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:52,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:52,281 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:52,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,282 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:52,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,284 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,284 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,284 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:52,284 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:52,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:52,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:52,284 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:52,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:52,284 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,284 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:52,285 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,285 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:52,285 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:52,285 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,285 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:52,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:52,327 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:52,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:52,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922492324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922492324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:52,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922492326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:52,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922492326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,385 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:52,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:52,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:52,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:52,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:52,390 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:52,390 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:52,391 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:52,391 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:52,391 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,391 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,393 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,393 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:52,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:52,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,393 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,393 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:52,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:52,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,394 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:52,395 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:52,395 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:52,395 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:52,395 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:52,396 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:52,396 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:52,396 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:52,396 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:52,396 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:52,399 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,399 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:52,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:52,399 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,399 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:52,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:52,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:52,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:52,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:52,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45eeb09a[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:52,401 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:52,401 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,402 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,402 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:52,402 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:52,402 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,403 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:52,403 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:52,404 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,404 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:52,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:52,404 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:52,404 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:52,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:52,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:52,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:52,409 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:52,409 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:52,409 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:52,409 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:52,409 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:52,411 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:52,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39470 deadline: 1680922492411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,414 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,414 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:52,414 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:52,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:52,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:52,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:52,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:52,415 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:52,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,419 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:52,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,421 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,421 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:52,421 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,421 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:52,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:52,421 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:52,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:52,421 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:52,423 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:52,423 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,423 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,423 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,423 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:52,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:52,423 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:52,423 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:52,523 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:52,524 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:52,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:52,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:52,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:52,532 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:52,532 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:52,536 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:52,537 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:52,537 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,537 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,540 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:52,540 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:52,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:52,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,542 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,542 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:52,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,542 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:52,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:52,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922492538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,543 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:52,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922492538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:52,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922492538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,543 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:52,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:52,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922492538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,548 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:52,548 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:52,548 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:52,550 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:52,550 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:52,550 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:52,550 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:52,550 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:52,551 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:52,551 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:52,551 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:52,551 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:52,551 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,551 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:52,551 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:52,557 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,557 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:52,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:52,558 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,558 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:52,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:52,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:52,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:52,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41bf7cef[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:52,561 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:52,561 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,562 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,562 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:52,562 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:52,562 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,562 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:52,562 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:52,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:52,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:52,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:52,565 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:52,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:52,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:52,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:52,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:52,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:52,570 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:52,570 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:52,570 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:52,570 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:52,570 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:52,570 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:52,573 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,573 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:52,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,573 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:52,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:52,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:52,577 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:52,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,579 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:52,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:52,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:52,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:52,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:52,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:52,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:52,582 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:52,585 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:52,585 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:52,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:52,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:52,647 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/41100221206d482091dfa1b977b233b8 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/41100221206d482091dfa1b977b233b8 2023-04-08 02:53:52,651 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/cd6570ec02584af483729deaa47f7ae4 2023-04-08 02:53:52,657 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/A of 9a2d8d09b1a5f9cc567f207bd50b4415 into 41100221206d482091dfa1b977b233b8(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:53:52,657 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:52,657 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/A, priority=13, startTime=1680922432153; duration=0sec 2023-04-08 02:53:52,657 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:53:52,682 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/652b796c688b45df9af805c1fe978dd4 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/652b796c688b45df9af805c1fe978dd4 2023-04-08 02:53:52,687 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:52,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:52,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:52,695 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/C of 9a2d8d09b1a5f9cc567f207bd50b4415 into 652b796c688b45df9af805c1fe978dd4(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:53:52,695 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:52,695 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/C, priority=13, startTime=1680922432155; duration=0sec 2023-04-08 02:53:52,695 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:53:52,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:52,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:52,697 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:52,697 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:52,698 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:52,698 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:52,700 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,700 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,701 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:52,701 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:52,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:52,702 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,702 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,702 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:52,702 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:52,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,703 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:52,705 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:52,705 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:52,705 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:52,705 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:52,708 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:52,709 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:52,709 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:52,709 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:52,709 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:52,709 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:52,709 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:52,709 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:52,709 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:52,709 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,709 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:52,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:52,711 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/e34ff3199c964bc7920db8ed8315703a 2023-04-08 02:53:52,717 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,717 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:52,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:52,718 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,718 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:52,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:52,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:52,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:52,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:52,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d8fef7c[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:52,720 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:52,720 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,723 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,723 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:52,723 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:52,723 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:52,724 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,724 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:52,725 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:52,725 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:52,725 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:52,726 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:52,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:52,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:52,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:52,743 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:52,743 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:52,743 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:52,743 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:52,744 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:52,748 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,749 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:52,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,749 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:52,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:52,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:52,750 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,753 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:52,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,766 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:52,767 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,767 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,767 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,767 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,767 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,767 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,767 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,767 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:52,767 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:52,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:52,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:52,767 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:52,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:52,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:52,768 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:52,768 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:52,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:52,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:52,845 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:52,845 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922492845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,850 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:52,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922492847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:52,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922492849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,851 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:52,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922492849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,868 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:52,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:52,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:52,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:52,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:52,875 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:52,875 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:52,875 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:52,875 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:52,876 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,876 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,877 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:52,877 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:52,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:52,878 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,878 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:52,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:52,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:52,884 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:52,884 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:52,884 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:52,884 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:52,884 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:52,888 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:52,888 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:52,888 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:52,888 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:52,888 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:52,888 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:52,888 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:52,888 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:52,888 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,888 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:52,888 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:52,890 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,890 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:52,890 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,890 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:52,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:52,893 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@12731379[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:52,893 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:52,893 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,895 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,895 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:52,895 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:52,895 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:52,896 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,896 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:52,897 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:52,897 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:52,897 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,897 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:52,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:52,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:52,899 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:52,899 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:52,899 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:52,900 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:52,900 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:52,902 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,902 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:52,902 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:52,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:52,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:52,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:52,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,903 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:52,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,904 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:52,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,914 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,914 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,914 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:52,914 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:52,914 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,914 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:52,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:52,914 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:52,914 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:52,914 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:52,914 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:52,914 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:52,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:52,914 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:52,914 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:52,915 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:52,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:52,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:52,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:53,015 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:53,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:53,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:53,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:53,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:53,019 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:53,019 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:53,019 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:53,019 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:53,020 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,020 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,021 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,021 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:53,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:53,021 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,021 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:53,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:53,023 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:53,023 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:53,023 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:53,023 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:53,023 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:53,023 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:53,023 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:53,023 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:53,023 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:53,023 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:53,023 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:53,023 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:53,023 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:53,023 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,023 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:53,023 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:53,026 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,026 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:53,027 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,027 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:53,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:53,028 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:53,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22a90bcd[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:53,029 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:53,029 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,030 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,030 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:53,030 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:53,030 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,030 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:53,030 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:53,031 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:53,031 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,031 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:53,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:53,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:53,031 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:53,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:53,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:53,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:53,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:53,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:53,034 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:53,034 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:53,034 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:53,034 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:53,034 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:53,036 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,036 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,036 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:53,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:53,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:53,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:53,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:53,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:53,037 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:53,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,038 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:53,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,041 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,041 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:53,041 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:53,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:53,041 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,041 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:53,042 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,042 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,042 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,042 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:53,042 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:53,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:53,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:53,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:53,043 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:53,043 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:53,143 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:53,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:53,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:53,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:53,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:53,148 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:53,148 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:53,148 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:53,148 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:53,148 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,148 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,149 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,149 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:53,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:53,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:53,150 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,150 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:53,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:53,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:53,151 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:53,151 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:53,151 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:53,151 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:53,151 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:53,152 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:53,152 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:53,152 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:53,152 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:53,152 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:53,152 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:53,152 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:53,152 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:53,152 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,152 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:53,152 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:53,154 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,154 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:53,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:53,154 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,154 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:53,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:53,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:53,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:53,156 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18cb5373[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:53,156 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:53,156 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,157 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,157 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,157 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:53,157 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:53,157 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,157 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:53,157 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:53,160 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:53,160 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:53,160 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:53,160 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:53,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:53,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:53,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:53,166 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:53,166 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:53,166 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:53,166 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:53,166 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:53,166 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:53,168 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,168 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,168 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:53,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:53,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:53,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:53,169 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,170 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:53,173 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:53,173 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:53,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:53,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:53,173 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:53,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:53,173 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:53,174 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:53,174 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:53,174 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,174 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,174 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,174 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,174 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,174 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,174 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,174 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:53,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:53,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:53,181 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=456 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/a0cff0586fa14d48a0fdea5cf970ce3d 2023-04-08 02:53:53,188 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/cd6570ec02584af483729deaa47f7ae4 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/cd6570ec02584af483729deaa47f7ae4 2023-04-08 02:53:53,195 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/cd6570ec02584af483729deaa47f7ae4, entries=200, sequenceid=456, filesize=14.3 K 2023-04-08 02:53:53,197 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/e34ff3199c964bc7920db8ed8315703a as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/e34ff3199c964bc7920db8ed8315703a 2023-04-08 02:53:53,204 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/e34ff3199c964bc7920db8ed8315703a, entries=150, sequenceid=456, filesize=11.9 K 2023-04-08 02:53:53,205 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/a0cff0586fa14d48a0fdea5cf970ce3d as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/a0cff0586fa14d48a0fdea5cf970ce3d 2023-04-08 02:53:53,215 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/a0cff0586fa14d48a0fdea5cf970ce3d, entries=150, sequenceid=456, filesize=11.9 K 2023-04-08 02:53:53,216 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 1016ms, sequenceid=456, compaction requested=true 2023-04-08 02:53:53,216 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:53,216 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:53:53,216 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-08 02:53:53,216 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:53:53,216 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:53:53,216 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:53:53,217 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-08 02:53:53,217 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-08 02:53:53,217 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. because compaction request was cancelled 2023-04-08 02:53:53,217 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-08 02:53:53,218 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37036 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:53:53,218 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/B is initiating minor compaction (all files) 2023-04-08 02:53:53,218 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/B in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:53,218 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/c3858b3e688e45e483b5b0de8835a937, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/ffea607b14e74d33838cd3974cb74f21, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/e34ff3199c964bc7920db8ed8315703a] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=36.2 K 2023-04-08 02:53:53,218 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-08 02:53:53,219 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-08 02:53:53,219 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. because compaction request was cancelled 2023-04-08 02:53:53,219 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting c3858b3e688e45e483b5b0de8835a937, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=405, earliestPutTs=1680922431144 2023-04-08 02:53:53,220 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting ffea607b14e74d33838cd3974cb74f21, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1680922431457 2023-04-08 02:53:53,221 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting e34ff3199c964bc7920db8ed8315703a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1680922431549 2023-04-08 02:53:53,233 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#B#compaction#53 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:53:53,255 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/4d3995b9fcb0422a80bef86c7e9d93f1 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/4d3995b9fcb0422a80bef86c7e9d93f1 2023-04-08 02:53:53,264 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/B of 9a2d8d09b1a5f9cc567f207bd50b4415 into 4d3995b9fcb0422a80bef86c7e9d93f1(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:53:53,264 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:53,264 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/B, priority=13, startTime=1680922433216; duration=0sec 2023-04-08 02:53:53,264 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:53:53,274 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:53,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:53,277 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:53,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:53,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:53,278 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:53,278 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:53,278 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:53,278 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:53,279 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,279 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,280 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:53,280 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:53,281 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,281 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,281 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,281 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:53,282 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:53,282 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:53,282 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:53,282 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:53,282 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:53,282 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:53,282 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:53,282 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:53,282 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:53,282 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-08 02:53:53,283 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:53:53,283 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:53,283 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:53:53,283 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:53,283 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:53:53,283 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:53,319 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/06ca2fc9d4d64c0a8befbace17c70bd2 2023-04-08 02:53:53,356 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:53,356 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/83e6157e3f8a45a093313490e95044dc 2023-04-08 02:53:53,357 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:53,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:53,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922493378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,382 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:53,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922493378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,382 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:53,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:39498 deadline: 1680922493379, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:53,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922493382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,394 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=473 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/81de0a9a3a36436bb52e1c178a5fb0d9 2023-04-08 02:53:53,402 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/06ca2fc9d4d64c0a8befbace17c70bd2 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/06ca2fc9d4d64c0a8befbace17c70bd2 2023-04-08 02:53:53,408 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/06ca2fc9d4d64c0a8befbace17c70bd2, entries=150, sequenceid=473, filesize=11.9 K 2023-04-08 02:53:53,409 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/83e6157e3f8a45a093313490e95044dc as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/83e6157e3f8a45a093313490e95044dc 2023-04-08 02:53:53,417 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/83e6157e3f8a45a093313490e95044dc, entries=150, sequenceid=473, filesize=11.9 K 2023-04-08 02:53:53,418 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/81de0a9a3a36436bb52e1c178a5fb0d9 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/81de0a9a3a36436bb52e1c178a5fb0d9 2023-04-08 02:53:53,429 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/81de0a9a3a36436bb52e1c178a5fb0d9, entries=150, sequenceid=473, filesize=11.9 K 2023-04-08 02:53:53,430 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 148ms, sequenceid=473, compaction requested=true 2023-04-08 02:53:53,430 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:53,430 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:53,430 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:53,437 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:53,437 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,437 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:53,437 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:53,441 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,441 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:53,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:53,441 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,441 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:53,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:53,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:53,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:53,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:53,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a6dab19[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:53,445 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:53,445 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,447 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,447 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:53,447 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:53,447 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:53,447 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,447 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:53,449 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:53,449 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:53,449 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,449 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:53,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:53,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:53,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:53,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:53,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:53,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:53,453 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:53,453 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:53,453 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:53,453 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:53,453 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:53,453 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:53,455 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,455 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:53,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,455 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:53,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:53,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:53,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:53,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:53,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,457 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:53,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:53,467 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,474 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,474 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:53,474 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:53,474 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:53,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:53,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:53,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:53,474 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:53,474 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,474 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,474 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,474 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,474 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,474 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:53,474 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:53,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:53,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:53,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:53,486 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-08 02:53:53,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922433371, max timestamp=1680922433485Num uniques 859; 2023-04-08 02:53:53,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680922433371, max timestamp=1680922433485Num uniques 859; 2023-04-08 02:53:53,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:53:53,489 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:53:53,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680922433371, max timestamp=1680922433485Num uniques 859; 2023-04-08 02:53:53,489 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:53:53,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-08 02:53:53,489 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:53:53,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:53:53,490 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:53:53,490 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:53:53,489 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-08 02:53:53,509 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/de4043d0d9444a80ab5a6d0a20118ceb 2023-04-08 02:53:53,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922433515, max timestamp=1680922433526Num uniques 859; 2023-04-08 02:53:53,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922433515, max timestamp=1680922433526Num uniques 859; 2023-04-08 02:53:53,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:53:53,528 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:53:53,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922433515, max timestamp=1680922433526Num uniques 859; 2023-04-08 02:53:53,529 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:53:53,543 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/a28c63f412c54690bea95ba60f243357 2023-04-08 02:53:53,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680922433535, max timestamp=1680922433566Num uniques 1018; 2023-04-08 02:53:53,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-08 02:53:53,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7680, totalHeapSize=17712, min timestamp=1680922433535, max timestamp=1680922433566Num uniques 1018; 2023-04-08 02:53:53,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680922433535, max timestamp=1680922433566Num uniques 1018; 2023-04-08 02:53:53,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46610, new segment data size=7680, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-08 02:53:53,569 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-08 02:53:53,575 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:53,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:53,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:53,601 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=494 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/db9959e359884f6db3e08dc9691c9989 2023-04-08 02:53:53,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:53,604 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:53,604 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:53,604 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:53,604 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:53,604 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:53,605 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,605 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,606 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=477, cellSize=21842, totalHeapSize=49848, min timestamp=1680922433566, max timestamp=1680922433604Num uniques 1027; 2023-04-08 02:53:53,607 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:53,607 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=21842, suffix heap size=134776,new segment heap size=49848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=477 2023-04-08 02:53:53,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:53,607 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:53,607 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,607 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:53,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=477, cellSize=21842, totalHeapSize=49848, min timestamp=1680922433566, max timestamp=1680922433604Num uniques 1027; 2023-04-08 02:53:53,609 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=21842, suffix heap size=134776,new segment heap size=49848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=477 2023-04-08 02:53:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:53,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,609 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:53,610 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:53,610 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:53,610 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:53,610 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:53,610 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:53,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=477, cellSize=21842, totalHeapSize=49848, min timestamp=1680922433566, max timestamp=1680922433604Num uniques 1027; 2023-04-08 02:53:53,611 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=21842, suffix heap size=134776,new segment heap size=49848 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=477 2023-04-08 02:53:53,612 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/de4043d0d9444a80ab5a6d0a20118ceb as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/de4043d0d9444a80ab5a6d0a20118ceb 2023-04-08 02:53:53,617 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:53,617 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:53,620 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:53,620 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:53,620 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:53,620 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:53,620 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:53,620 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:53,620 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/de4043d0d9444a80ab5a6d0a20118ceb, entries=200, sequenceid=494, filesize=14.3 K 2023-04-08 02:53:53,620 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,621 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:53,621 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:53,622 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/a28c63f412c54690bea95ba60f243357 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/a28c63f412c54690bea95ba60f243357 2023-04-08 02:53:53,624 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,624 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:53,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:53,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:53,625 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,625 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:53,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:53,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:53,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@408f91f6[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:53,627 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:53,627 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,628 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,628 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:53,628 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:53,628 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:53,629 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,629 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:53,630 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:53,630 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:53,630 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:53,630 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:53,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:53,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:53,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:53,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:53,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:53,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:53,634 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:53,635 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:53,635 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:53,635 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:53,635 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:53,636 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/a28c63f412c54690bea95ba60f243357, entries=150, sequenceid=494, filesize=11.9 K 2023-04-08 02:53:53,636 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,637 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,637 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:53,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:53,637 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:53,638 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/db9959e359884f6db3e08dc9691c9989 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/db9959e359884f6db3e08dc9691c9989 2023-04-08 02:53:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:53,639 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:53,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,643 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:53,643 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:53,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:53,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:53,643 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,643 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,644 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,644 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,644 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,644 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,644 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:53,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:53,644 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:53,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:53,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:53,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:53,644 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:53,645 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:53,652 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/db9959e359884f6db3e08dc9691c9989, entries=150, sequenceid=494, filesize=11.9 K 2023-04-08 02:53:53,653 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~31.31 KB/32060, heapSize ~76.34 KB/78176, currentSize=161.02 KB/164880 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 167ms, sequenceid=494, compaction requested=true 2023-04-08 02:53:53,654 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:53,654 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:53:53,654 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:53:53,654 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:53:53,654 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:53:53,654 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:53:53,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:53,655 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=174.43 KB heapSize=474.70 KB 2023-04-08 02:53:53,656 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:53:53,656 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:53:53,656 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:53:53,656 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:53:53,656 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:53:53,656 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:53:53,657 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37138 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:53:53,657 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/B is initiating minor compaction (all files) 2023-04-08 02:53:53,657 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/B in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:53,657 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54170 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:53:53,657 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/4d3995b9fcb0422a80bef86c7e9d93f1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/83e6157e3f8a45a093313490e95044dc, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/a28c63f412c54690bea95ba60f243357] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=36.3 K 2023-04-08 02:53:53,657 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/A is initiating minor compaction (all files) 2023-04-08 02:53:53,657 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/A in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:53,658 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/41100221206d482091dfa1b977b233b8, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/cd6570ec02584af483729deaa47f7ae4, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/06ca2fc9d4d64c0a8befbace17c70bd2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/de4043d0d9444a80ab5a6d0a20118ceb] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=52.9 K 2023-04-08 02:53:53,659 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 41100221206d482091dfa1b977b233b8, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1680922431457 2023-04-08 02:53:53,659 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 4d3995b9fcb0422a80bef86c7e9d93f1, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1680922431549 2023-04-08 02:53:53,670 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting cd6570ec02584af483729deaa47f7ae4, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1680922431549 2023-04-08 02:53:53,670 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 83e6157e3f8a45a093313490e95044dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1680922432211 2023-04-08 02:53:53,671 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 06ca2fc9d4d64c0a8befbace17c70bd2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1680922432211 2023-04-08 02:53:53,672 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting a28c63f412c54690bea95ba60f243357, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1680922433371 2023-04-08 02:53:53,674 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting de4043d0d9444a80ab5a6d0a20118ceb, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1680922433371 2023-04-08 02:53:53,675 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:53,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922493656, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:53,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922493673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,680 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:53,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922493676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,685 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:53,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922493677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,699 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#A#compaction#61 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:53:53,713 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#B#compaction#62 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:53:53,741 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/088d6a9f1f8d47c49f62e84a7fbbb004 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/088d6a9f1f8d47c49f62e84a7fbbb004 2023-04-08 02:53:53,745 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:53,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:53,749 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/B of 9a2d8d09b1a5f9cc567f207bd50b4415 into 088d6a9f1f8d47c49f62e84a7fbbb004(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:53:53,749 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:53,749 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/B, priority=13, startTime=1680922433654; duration=0sec 2023-04-08 02:53:53,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:53,749 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:53:53,749 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:53:53,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:53,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:53,751 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:53,751 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:53,751 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:53,751 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:53,751 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49290 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:53:53,751 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,751 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/C is initiating minor compaction (all files) 2023-04-08 02:53:53,751 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,751 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/C in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:53,752 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/652b796c688b45df9af805c1fe978dd4, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/a0cff0586fa14d48a0fdea5cf970ce3d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/81de0a9a3a36436bb52e1c178a5fb0d9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/db9959e359884f6db3e08dc9691c9989] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=48.1 K 2023-04-08 02:53:53,752 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 652b796c688b45df9af805c1fe978dd4, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1680922431457 2023-04-08 02:53:53,753 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting a0cff0586fa14d48a0fdea5cf970ce3d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=456, earliestPutTs=1680922431549 2023-04-08 02:53:53,753 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,753 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:53,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:53,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:53,759 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 81de0a9a3a36436bb52e1c178a5fb0d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=473, earliestPutTs=1680922432211 2023-04-08 02:53:53,759 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,759 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:53,759 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting db9959e359884f6db3e08dc9691c9989, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1680922433371 2023-04-08 02:53:53,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:53,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:53,764 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:53,764 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:53,764 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:53,764 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:53,764 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:53,766 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:53,766 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:53,767 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:53,767 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:53,767 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:53,767 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:53,767 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:53,767 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:53,767 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,767 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:53,767 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:53,769 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:53,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:53,770 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:53,771 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,771 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:53,771 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#C#compaction#63 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:53:53,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:53,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:53,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a120d49[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:53,773 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:53,773 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,775 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,775 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,775 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:53,775 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:53,775 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:53,775 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:53,777 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:53,777 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:53,777 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:53,777 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,777 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:53,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:53,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922493777, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:53,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:53,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:53,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:53,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:53,781 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:53,781 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:53,781 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:53,781 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:53,781 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:53,785 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,785 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:53,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:53,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,785 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:53,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:53,786 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:53,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:53,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,787 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:53,787 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922493782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:53,788 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:53,788 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922493787, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,788 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:53,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,790 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:53,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922493789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,792 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,792 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,792 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,792 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,792 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:53,792 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:53,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:53,792 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:53,792 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,792 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,793 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:53,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:53,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:53,793 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:53,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:53,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:53,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:53,793 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:53,893 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:53,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:53,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:53,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:53,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:53,900 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:53,900 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:53,900 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:53,900 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:53,901 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,901 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,902 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:53,902 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:53,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:53,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:53,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,904 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:53,904 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,904 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:53,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:53,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:53,908 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:53,908 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:53,908 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:53,908 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:53,908 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:53,909 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:53,909 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:53,909 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:53,910 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:53,910 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:53,910 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:53,910 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,910 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:53,910 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:53,913 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,913 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:53,913 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,913 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:53,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:53,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:53,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@680093b3[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:53,915 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:53,915 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,916 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,917 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:53,917 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:53,917 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,917 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:53,917 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:53,920 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,920 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:53,920 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:53,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:53,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:53,920 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:53,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:53,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:53,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:53,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:53,923 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:53,923 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:53,923 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:53,923 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:53,923 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:53,925 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,925 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:53,925 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:53,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:53,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:53,926 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:53,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,928 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:53,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:53,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,932 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:53,932 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,932 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:53,932 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:53,932 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:53,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:53,932 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:53,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:53,932 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,932 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:53,932 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,932 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,932 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:53,932 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:53,932 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:53,932 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:53,932 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:53,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:53,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:53,984 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:53,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922493980, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,991 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:53,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922493988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:53,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922493990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:53,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:53,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922493993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,032 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:54,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:54,036 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:54,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:54,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:54,037 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:54,037 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:54,037 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:54,037 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:54,038 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,038 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,039 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:54,039 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:54,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:54,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,040 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,040 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:54,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:54,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,041 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:54,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:54,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:54,041 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:54,041 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:54,041 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:54,042 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:54,042 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:54,042 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:54,042 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:54,042 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:54,042 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:54,042 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:54,042 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:54,042 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,042 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:54,042 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:54,047 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:54,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:54,047 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:54,048 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,048 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:54,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:54,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:54,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:54,050 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f12c8c9[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:54,050 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:54,050 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,051 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,052 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:54,052 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:54,052 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:54,052 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,052 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:54,053 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:54,054 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:54,054 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:54,054 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:54,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:54,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:54,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:54,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:54,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:54,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:54,057 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:54,057 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:54,057 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:54,057 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:54,057 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:54,059 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,059 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,059 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:54,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:54,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:54,060 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:54,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,064 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:54,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,068 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:54,068 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:54,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:54,068 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,068 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:54,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:54,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:54,068 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:54,068 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,069 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,068 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:54,069 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:54,069 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:54,069 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,069 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,069 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,069 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:54,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:54,128 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=565 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/9a7c7ffceca4474caf5a8816c62b1bef 2023-04-08 02:53:54,142 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/a391d164dd7f4ceaac2017e6451bba5e as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/a391d164dd7f4ceaac2017e6451bba5e 2023-04-08 02:53:54,159 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/A of 9a2d8d09b1a5f9cc567f207bd50b4415 into a391d164dd7f4ceaac2017e6451bba5e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:53:54,159 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:54,159 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/A, priority=12, startTime=1680922433654; duration=0sec 2023-04-08 02:53:54,159 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:53:54,169 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:54,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:54,174 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:54,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:54,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:54,193 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:54,193 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:54,193 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:54,193 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:54,195 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,195 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,196 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:54,196 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,197 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:54,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:54,197 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,197 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:54,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,198 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:54,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,199 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:54,200 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:54,200 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:54,200 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:54,200 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:54,200 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:54,202 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/914d60c335c84bdbaf05d3ae29c36994 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/914d60c335c84bdbaf05d3ae29c36994 2023-04-08 02:53:54,208 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:54,208 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/C of 9a2d8d09b1a5f9cc567f207bd50b4415 into 914d60c335c84bdbaf05d3ae29c36994(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:53:54,208 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:54,209 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:54,209 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=565 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/4ac37fa2ad874e92b8cccb0e52f96d39 2023-04-08 02:53:54,209 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/C, priority=12, startTime=1680922433654; duration=0sec 2023-04-08 02:53:54,209 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:53:54,210 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:54,212 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:54,212 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:54,212 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:54,212 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:54,212 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:54,212 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,216 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:54,216 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:54,217 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:54,218 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:54,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:54,219 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,219 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:54,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:54,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:54,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@732ee0fc[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:54,221 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:54,222 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,224 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,224 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:54,225 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,225 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:54,225 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:54,225 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:54,228 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,228 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:54,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:54,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:54,229 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:54,229 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:54,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:54,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:54,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:54,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:54,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:54,233 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:54,233 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:54,233 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:54,233 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:54,233 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:54,235 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,235 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:54,235 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:54,235 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:54,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,238 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,241 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:54,241 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,241 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,241 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,241 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,241 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,241 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,241 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:54,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:54,241 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:54,241 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:54,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:54,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:54,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:54,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:54,242 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:54,242 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:54,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:54,260 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=58.14 KB at sequenceid=565 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/0a323285bf2b49e384581a100f36e5f6 2023-04-08 02:53:54,269 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/9a7c7ffceca4474caf5a8816c62b1bef as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/9a7c7ffceca4474caf5a8816c62b1bef 2023-04-08 02:53:54,277 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/9a7c7ffceca4474caf5a8816c62b1bef, entries=350, sequenceid=565, filesize=21.4 K 2023-04-08 02:53:54,278 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/4ac37fa2ad874e92b8cccb0e52f96d39 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/4ac37fa2ad874e92b8cccb0e52f96d39 2023-04-08 02:53:54,285 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/4ac37fa2ad874e92b8cccb0e52f96d39, entries=150, sequenceid=565, filesize=11.9 K 2023-04-08 02:53:54,287 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/0a323285bf2b49e384581a100f36e5f6 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/0a323285bf2b49e384581a100f36e5f6 2023-04-08 02:53:54,291 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:54,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922494287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,293 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/0a323285bf2b49e384581a100f36e5f6, entries=150, sequenceid=565, filesize=11.9 K 2023-04-08 02:53:54,294 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~174.43 KB/178620, heapSize ~474.66 KB/486048, currentSize=20.13 KB/20610 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 639ms, sequenceid=565, compaction requested=false 2023-04-08 02:53:54,295 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:54,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:54,314 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-08 02:53:54,314 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:53:54,314 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:54,314 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:53:54,314 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:54,314 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:53:54,315 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:54,342 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:54,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:54,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:54,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:54,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:54,359 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:54,359 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:54,359 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:54,359 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:54,363 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,363 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,365 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:54,365 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:54,365 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,365 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:54,366 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:54,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,366 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:54,373 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:54,373 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:54,373 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:54,373 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:54,373 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:54,374 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/e1d948bc142642d8a5b019b0f73cea95 2023-04-08 02:53:54,376 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:54,376 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:54,376 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:54,376 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:54,376 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:54,376 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:54,377 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:54,377 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:54,377 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,377 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:54,377 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:54,378 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:54,379 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:54,381 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,381 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:54,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:54,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:54,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:54,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49ecd395[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:54,384 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:54,385 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,387 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,388 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:54,388 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:54,388 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:54,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922434373, max timestamp=1680922434385Num uniques 859; 2023-04-08 02:53:54,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:53:54,389 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922434373, max timestamp=1680922434385Num uniques 859; 2023-04-08 02:53:54,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:53:54,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922434373, max timestamp=1680922434385Num uniques 859; 2023-04-08 02:53:54,390 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:53:54,390 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,390 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:54,391 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:54,391 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:54,391 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,391 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:54,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:54,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:54,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:54,401 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:54,401 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:54,401 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:54,401 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:54,401 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:54,403 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:54,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:54,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:54,404 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,404 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:54,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,404 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,406 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,409 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,409 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:54,409 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:54,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:54,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:54,409 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:54,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:54,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:54,409 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,409 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,409 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,409 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,409 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,409 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,409 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:54,409 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:54,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:54,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:54,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:54,420 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/5f3da0869d02414db61448d84c2afb69 2023-04-08 02:53:54,431 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:54,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 224 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922494424, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,439 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:54,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 179 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922494428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,439 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:54,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922494431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,440 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:54,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922494435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,458 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/9e757487ae2542059992d6c55e562919 2023-04-08 02:53:54,467 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/e1d948bc142642d8a5b019b0f73cea95 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/e1d948bc142642d8a5b019b0f73cea95 2023-04-08 02:53:54,474 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/e1d948bc142642d8a5b019b0f73cea95, entries=200, sequenceid=578, filesize=14.3 K 2023-04-08 02:53:54,476 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/5f3da0869d02414db61448d84c2afb69 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/5f3da0869d02414db61448d84c2afb69 2023-04-08 02:53:54,488 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/5f3da0869d02414db61448d84c2afb69, entries=150, sequenceid=578, filesize=11.9 K 2023-04-08 02:53:54,490 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/9e757487ae2542059992d6c55e562919 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/9e757487ae2542059992d6c55e562919 2023-04-08 02:53:54,510 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:54,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:54,515 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:54,517 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/9e757487ae2542059992d6c55e562919, entries=150, sequenceid=578, filesize=11.9 K 2023-04-08 02:53:54,520 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 206ms, sequenceid=578, compaction requested=true 2023-04-08 02:53:54,521 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:54,521 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:53:54,521 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:53:54,521 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:53:54,521 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:53:54,522 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:53:54,523 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:53:54,523 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/B is initiating minor compaction (all files) 2023-04-08 02:53:54,523 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/B in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:54,524 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/088d6a9f1f8d47c49f62e84a7fbbb004, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/4ac37fa2ad874e92b8cccb0e52f96d39, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/5f3da0869d02414db61448d84c2afb69] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=36.4 K 2023-04-08 02:53:54,524 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 49444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:53:54,525 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/A is initiating minor compaction (all files) 2023-04-08 02:53:54,525 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/A in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:54,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:54,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:54,525 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:54,525 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:54,525 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:54,525 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:54,525 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/a391d164dd7f4ceaac2017e6451bba5e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/9a7c7ffceca4474caf5a8816c62b1bef, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/e1d948bc142642d8a5b019b0f73cea95] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=48.3 K 2023-04-08 02:53:54,525 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 088d6a9f1f8d47c49f62e84a7fbbb004, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1680922433371 2023-04-08 02:53:54,526 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,526 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,526 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 4ac37fa2ad874e92b8cccb0e52f96d39, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=565, earliestPutTs=1680922433633 2023-04-08 02:53:54,527 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 5f3da0869d02414db61448d84c2afb69, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1680922434306 2023-04-08 02:53:54,527 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting a391d164dd7f4ceaac2017e6451bba5e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1680922433371 2023-04-08 02:53:54,528 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,528 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:54,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:54,529 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,529 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:54,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,530 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 9a7c7ffceca4474caf5a8816c62b1bef, keycount=350, bloomtype=ROW, size=21.4 K, encoding=NONE, compression=NONE, seqNum=565, earliestPutTs=1680922433625 2023-04-08 02:53:54,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,530 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting e1d948bc142642d8a5b019b0f73cea95, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1680922433671 2023-04-08 02:53:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:54,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,530 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:54,540 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:54,540 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:54,540 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:54,540 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:54,540 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:54,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680922434420, max timestamp=1680922434538Num uniques 1018; 2023-04-08 02:53:54,544 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-08 02:53:54,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:54,545 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=102.87 KB heapSize=292.20 KB 2023-04-08 02:53:54,546 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:53:54,546 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:53:54,546 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:53:54,546 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-08 02:53:54,546 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:53:54,546 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-08 02:53:54,552 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:54,552 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:54,552 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:54,552 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:54,552 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:54,552 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:54,552 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:54,552 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:54,552 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,552 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:54,553 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:54,553 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#B#compaction#69 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:53:54,553 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#A#compaction#70 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:53:54,555 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,555 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:54,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:54,555 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:54,555 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:54,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:54,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:54,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:54,558 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@251d5c10[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:54,558 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:54,558 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,563 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:54,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:54,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:54,565 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,565 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:54,568 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:54,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:54,568 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:54,568 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:54,568 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:54,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:54,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:54,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:54,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:54,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:54,573 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:54,573 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:54,573 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:54,573 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:54,573 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:54,575 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,575 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,575 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:54,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:54,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:54,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:54,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:54,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:54,577 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:54,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,580 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:54,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,584 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:54,584 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:54,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:54,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:54,585 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,585 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:54,585 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:54,585 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:54,585 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,585 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,585 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,585 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,585 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,585 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:54,585 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:54,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:54,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:54,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:54,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922494577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,586 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:54,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922494579, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:54,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922494580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,589 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:54,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922494583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,633 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=616 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/93b0b7a4dac34519bc0900b5a8880c40 2023-04-08 02:53:54,640 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/e890df3a7eda4aeaa6ea5563e0d966c1 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/e890df3a7eda4aeaa6ea5563e0d966c1 2023-04-08 02:53:54,646 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/f7fcc74134604b59aec41abc6a122fc8 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/f7fcc74134604b59aec41abc6a122fc8 2023-04-08 02:53:54,658 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/A of 9a2d8d09b1a5f9cc567f207bd50b4415 into e890df3a7eda4aeaa6ea5563e0d966c1(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:53:54,658 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:54,658 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/A, priority=13, startTime=1680922434521; duration=0sec 2023-04-08 02:53:54,659 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:53:54,659 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:53:54,665 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37240 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:53:54,666 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/C is initiating minor compaction (all files) 2023-04-08 02:53:54,666 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/C in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:54,666 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/914d60c335c84bdbaf05d3ae29c36994, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/0a323285bf2b49e384581a100f36e5f6, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/9e757487ae2542059992d6c55e562919] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=36.4 K 2023-04-08 02:53:54,667 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 914d60c335c84bdbaf05d3ae29c36994, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=494, earliestPutTs=1680922433371 2023-04-08 02:53:54,669 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 0a323285bf2b49e384581a100f36e5f6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=565, earliestPutTs=1680922433633 2023-04-08 02:53:54,670 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 9e757487ae2542059992d6c55e562919, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1680922434306 2023-04-08 02:53:54,671 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/B of 9a2d8d09b1a5f9cc567f207bd50b4415 into f7fcc74134604b59aec41abc6a122fc8(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:53:54,671 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:54,671 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/B, priority=13, startTime=1680922434521; duration=0sec 2023-04-08 02:53:54,671 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:53:54,685 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#C#compaction#73 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:53:54,685 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:54,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:54,687 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=616 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/230365b62df54c64908c57482813f7c2 2023-04-08 02:53:54,689 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:54,689 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:54,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:54,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922494688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922494688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,690 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:54,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922494690, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,695 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:54,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922494693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:54,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:54,704 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:54,704 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:54,704 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:54,704 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:54,705 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,705 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,707 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,707 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:54,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:54,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:54,708 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,708 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:54,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:54,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:54,720 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:54,720 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:54,720 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:54,720 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:54,720 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:54,723 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:54,723 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:54,723 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:54,723 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:54,723 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:54,723 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:54,723 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:54,723 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:54,723 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,726 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:54,726 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:54,728 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,728 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:54,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:54,729 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,729 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:54,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:54,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:54,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:54,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:54,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c909cfa[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:54,732 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:54,732 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,734 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,734 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:54,734 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:54,734 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:54,736 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:54,736 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:54,736 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:54,736 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/a5355e7e85a64193aac1abcb12dc1b0c as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/a5355e7e85a64193aac1abcb12dc1b0c 2023-04-08 02:53:54,739 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,739 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:54,739 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:54,740 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:54,740 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:54,740 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:54,740 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:54,740 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:54,743 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,743 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:54,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:54,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:54,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,744 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:54,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:54,745 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/C of 9a2d8d09b1a5f9cc567f207bd50b4415 into a5355e7e85a64193aac1abcb12dc1b0c(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:53:54,745 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:54,745 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/C, priority=13, startTime=1680922434522; duration=0sec 2023-04-08 02:53:54,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:54,745 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:53:54,745 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:54,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:54,749 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,752 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,752 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,752 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,752 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:54,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:54,752 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,752 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,752 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,752 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:54,752 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:54,752 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:54,753 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:54,753 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:54,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:54,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:54,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:54,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:54,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922494798, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,852 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:54,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:54,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:54,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:54,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:54,857 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:54,857 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:54,857 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:54,857 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:54,857 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,857 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,860 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:54,860 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:54,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:54,861 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,861 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:54,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:54,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,862 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:54,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:54,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:54,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:54,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:54,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:54,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:54,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:54,862 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:54,863 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:54,863 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:54,863 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:54,863 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:54,863 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:54,863 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,863 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:54,863 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:54,864 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,864 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:54,865 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,865 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:54,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:54,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:54,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78bdba94[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:54,867 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:54,867 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,868 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,868 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:54,868 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:54,868 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:54,871 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,872 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:54,874 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:54,874 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:54,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:54,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:54,878 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:54,878 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:54,878 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:54,878 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:54,878 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:54,880 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,880 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:54,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,881 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:54,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:54,882 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:54,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:54,885 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:54,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,887 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:54,887 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:54,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:54,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:54,888 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:54,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:54,888 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,889 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,889 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,889 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,889 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:54,889 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,889 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:54,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:54,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:54,890 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:54,890 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:54,893 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:54,893 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:54,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922494893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922494893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:54,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922494894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,898 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:54,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922494896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,990 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:54,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:54,993 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:54,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:54,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:54,994 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:54,994 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:54,995 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:54,995 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:54,995 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,995 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,996 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:54,996 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:54,997 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,997 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:54,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,997 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:54,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:54,998 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:54,998 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:54,998 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:54,998 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:54,998 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:54,998 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:54,999 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:54,999 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:54,999 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:54,999 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:54,999 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:54,999 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:54,999 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:54,999 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:54,999 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:54,999 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:54,999 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:55,001 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,001 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:55,002 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,002 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:55,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:55,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:55,004 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5762be6d[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:55,004 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:55,004 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,005 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,005 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,006 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:55,006 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:55,006 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:55,006 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,006 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:55,010 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:55,010 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:55,010 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:55,010 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:55,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:55,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:55,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:55,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:55,015 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:55,015 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:55,015 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:55,015 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:55,015 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:55,019 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:55,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:55,019 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,019 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:55,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:55,020 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:55,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:55,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,021 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:55,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:55,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:55,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,024 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:55,024 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:55,024 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,024 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:55,024 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:55,024 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,024 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,024 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,024 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:55,024 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:55,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:55,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:55,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:55,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:55,025 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:55,025 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:55,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:55,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:55,025 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:55,125 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:55,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:55,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:55,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:55,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:55,129 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:55,129 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:55,129 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:55,129 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:55,129 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,129 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:55,131 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:55,131 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:55,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:55,131 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,131 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:55,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:55,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:55,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:55,132 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:55,132 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:55,132 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:55,132 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:55,132 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:55,132 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:55,133 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:55,133 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:55,133 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:55,133 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:55,134 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:55,134 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:55,134 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:55,134 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:55,134 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,134 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:55,134 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:55,135 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,135 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:55,135 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,135 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:55,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:55,137 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18d0854d[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:55,137 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:55,137 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,139 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,139 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,139 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:55,139 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:55,139 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:55,139 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,139 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:55,140 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,140 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:55,140 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:55,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:55,140 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:55,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:55,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:55,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:55,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:55,143 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:55,143 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:55,143 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:55,143 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:55,143 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:55,144 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,144 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,144 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:55,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:55,145 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:55,147 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:55,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,149 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,149 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:55,149 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:55,149 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:55,149 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:55,149 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:55,149 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:55,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:55,149 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:55,149 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,149 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,149 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,149 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:55,149 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:55,149 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:55,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:55,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:55,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:55,179 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=616 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/2c4fa4b2355740fc9456df420f6897f1 2023-04-08 02:53:55,186 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/93b0b7a4dac34519bc0900b5a8880c40 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/93b0b7a4dac34519bc0900b5a8880c40 2023-04-08 02:53:55,192 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/93b0b7a4dac34519bc0900b5a8880c40, entries=150, sequenceid=616, filesize=11.9 K 2023-04-08 02:53:55,193 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/230365b62df54c64908c57482813f7c2 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/230365b62df54c64908c57482813f7c2 2023-04-08 02:53:55,197 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:55,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922495196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,198 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:55,198 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922495197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,199 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:55,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922495197, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,202 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/230365b62df54c64908c57482813f7c2, entries=150, sequenceid=616, filesize=11.9 K 2023-04-08 02:53:55,203 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/2c4fa4b2355740fc9456df420f6897f1 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/2c4fa4b2355740fc9456df420f6897f1 2023-04-08 02:53:55,204 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:55,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922495201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,209 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/2c4fa4b2355740fc9456df420f6897f1, entries=150, sequenceid=616, filesize=11.9 K 2023-04-08 02:53:55,210 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~102.87 KB/105340, heapSize ~292.16 KB/299168, currentSize=73.80 KB/75570 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 665ms, sequenceid=616, compaction requested=false 2023-04-08 02:53:55,210 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:55,250 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:55,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:55,252 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:55,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:55,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:55,254 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:55,254 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:55,254 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:55,254 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:55,254 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,254 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:55,257 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:55,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:55,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:55,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:55,257 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,258 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,258 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,258 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:55,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:55,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:55,258 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:55,259 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:55,259 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:55,259 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:55,259 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:55,259 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:55,260 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:55,260 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:55,260 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:55,260 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:55,260 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-08 02:53:55,260 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:53:55,260 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:55,261 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:53:55,261 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:55,261 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:53:55,261 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:55,674 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=633 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/4abfff4b5b434458afab50099f879917 2023-04-08 02:53:55,693 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=633 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/2483fee87e024cc785f904ff5525d371 2023-04-08 02:53:55,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:55,702 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:55,726 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=633 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/a6147ebfa0594798b6e649b570fd6947 2023-04-08 02:53:55,732 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/4abfff4b5b434458afab50099f879917 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/4abfff4b5b434458afab50099f879917 2023-04-08 02:53:55,737 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/4abfff4b5b434458afab50099f879917, entries=150, sequenceid=633, filesize=11.9 K 2023-04-08 02:53:55,738 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/2483fee87e024cc785f904ff5525d371 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/2483fee87e024cc785f904ff5525d371 2023-04-08 02:53:55,751 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/2483fee87e024cc785f904ff5525d371, entries=150, sequenceid=633, filesize=11.9 K 2023-04-08 02:53:55,753 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/a6147ebfa0594798b6e649b570fd6947 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/a6147ebfa0594798b6e649b570fd6947 2023-04-08 02:53:55,753 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:55,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922495749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,753 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:55,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922495751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:55,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922495752, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:55,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922495754, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,760 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/a6147ebfa0594798b6e649b570fd6947, entries=150, sequenceid=633, filesize=11.9 K 2023-04-08 02:53:55,761 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 501ms, sequenceid=633, compaction requested=true 2023-04-08 02:53:55,761 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:55,761 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:55,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:55,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:55,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,764 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:55,764 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:55,766 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,766 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:55,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:55,767 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,767 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:55,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:55,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:55,770 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@605fade1[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:55,770 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:55,770 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,771 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,771 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:55,771 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:55,771 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:55,771 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,771 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:55,773 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:55,773 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,773 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:55,773 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:55,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:55,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:55,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:55,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:55,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:55,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:55,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:55,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:55,778 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:55,778 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:55,778 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:55,778 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:55,778 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:55,778 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:55,781 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,781 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:55,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:55,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:55,782 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:55,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:55,784 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:55,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:55,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,789 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:55,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:55,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,795 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:55,795 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,795 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:55,795 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:55,795 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,795 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,795 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,795 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:55,795 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:55,795 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:55,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:55,795 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:55,796 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:55,796 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:55,796 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:55,796 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:55,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:55,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:55,815 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-08 02:53:55,815 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:53:55,816 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:55,816 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:53:55,816 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:55,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:55,816 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:53:55,816 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:55,864 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:55,864 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:55,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922495860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,864 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922495860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:55,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922495862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:55,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922495862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:55,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922495864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,874 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=652 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/42b03fff9cf24ee6a6ba18ffbf0bcee9 2023-04-08 02:53:55,895 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=652 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/76969ff8ecfc495db08ba6f7400d3995 2023-04-08 02:53:55,897 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:55,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:55,900 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:55,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:55,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:55,903 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:55,903 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:55,904 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:55,904 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:55,906 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,906 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:55,907 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:55,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:55,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:55,908 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:55,909 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,909 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:55,909 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:55,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:55,909 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:55,910 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:55,910 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:55,910 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:55,910 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:55,912 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:55,912 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:55,912 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:55,912 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:55,912 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:55,912 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:55,912 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:55,912 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:55,912 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,918 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:55,918 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:55,920 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:55,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:55,920 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,921 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,921 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:55,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:55,922 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:55,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@75a3d264[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:55,923 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:55,923 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,924 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,924 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:55,924 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:55,924 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:55,925 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,925 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:55,925 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=652 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/35d50f4578d34430a8bab026cd78fbd2 2023-04-08 02:53:55,926 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:55,926 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:55,926 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:55,926 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:55,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:55,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:55,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:55,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:55,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,932 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/42b03fff9cf24ee6a6ba18ffbf0bcee9 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/42b03fff9cf24ee6a6ba18ffbf0bcee9 2023-04-08 02:53:55,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:55,932 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:55,932 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:55,933 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:55,933 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:55,933 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:55,933 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:55,936 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,936 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:55,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:55,937 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:55,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:55,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:55,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,938 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:55,939 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/42b03fff9cf24ee6a6ba18ffbf0bcee9, entries=200, sequenceid=652, filesize=14.3 K 2023-04-08 02:53:55,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:55,940 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/76969ff8ecfc495db08ba6f7400d3995 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/76969ff8ecfc495db08ba6f7400d3995 2023-04-08 02:53:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:55,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:55,941 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:55,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:55,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,947 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:55,947 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:55,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:55,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:55,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:55,948 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,948 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:55,948 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:55,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:55,948 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:55,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:55,948 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:55,948 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:55,949 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,949 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:55,949 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:55,949 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:55,949 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:55,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:55,949 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/76969ff8ecfc495db08ba6f7400d3995, entries=150, sequenceid=652, filesize=11.9 K 2023-04-08 02:53:55,950 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/35d50f4578d34430a8bab026cd78fbd2 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/35d50f4578d34430a8bab026cd78fbd2 2023-04-08 02:53:55,957 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/35d50f4578d34430a8bab026cd78fbd2, entries=150, sequenceid=652, filesize=11.9 K 2023-04-08 02:53:55,959 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 144ms, sequenceid=652, compaction requested=true 2023-04-08 02:53:55,959 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:55,959 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:53:55,959 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:53:55,959 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:53:55,959 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:53:55,959 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:53:55,961 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 51968 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:53:55,961 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:53:55,961 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/A is initiating minor compaction (all files) 2023-04-08 02:53:55,961 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/B is initiating minor compaction (all files) 2023-04-08 02:53:55,961 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/A in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:55,961 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/B in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:55,961 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/e890df3a7eda4aeaa6ea5563e0d966c1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/93b0b7a4dac34519bc0900b5a8880c40, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/4abfff4b5b434458afab50099f879917, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/42b03fff9cf24ee6a6ba18ffbf0bcee9] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=50.8 K 2023-04-08 02:53:55,961 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/f7fcc74134604b59aec41abc6a122fc8, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/230365b62df54c64908c57482813f7c2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/2483fee87e024cc785f904ff5525d371, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/76969ff8ecfc495db08ba6f7400d3995] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=48.4 K 2023-04-08 02:53:55,962 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting e890df3a7eda4aeaa6ea5563e0d966c1, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1680922434306 2023-04-08 02:53:55,962 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting f7fcc74134604b59aec41abc6a122fc8, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1680922434306 2023-04-08 02:53:55,963 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 93b0b7a4dac34519bc0900b5a8880c40, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=616, earliestPutTs=1680922434423 2023-04-08 02:53:55,963 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 230365b62df54c64908c57482813f7c2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=616, earliestPutTs=1680922434423 2023-04-08 02:53:55,964 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 4abfff4b5b434458afab50099f879917, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=633, earliestPutTs=1680922434577 2023-04-08 02:53:55,964 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 2483fee87e024cc785f904ff5525d371, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=633, earliestPutTs=1680922434577 2023-04-08 02:53:55,965 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 42b03fff9cf24ee6a6ba18ffbf0bcee9, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=652, earliestPutTs=1680922435747 2023-04-08 02:53:55,965 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 76969ff8ecfc495db08ba6f7400d3995, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=652, earliestPutTs=1680922435747 2023-04-08 02:53:55,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:55,972 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-08 02:53:55,973 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:53:55,973 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:55,973 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:53:55,973 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:55,973 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:53:55,973 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:55,983 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#B#compaction#81 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:53:55,990 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#A#compaction#82 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:53:56,033 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:56,033 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922496003, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,036 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:56,036 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:56,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922496029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:56,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922496032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,037 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:56,037 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922496033, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922496029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,044 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/b33bafe0044c48dabe3acce857aae963 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/b33bafe0044c48dabe3acce857aae963 2023-04-08 02:53:56,048 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/0f6bacf9c35143d5b8dcdb050ee6adc6 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/0f6bacf9c35143d5b8dcdb050ee6adc6 2023-04-08 02:53:56,049 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:56,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:56,053 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:56,054 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/B of 9a2d8d09b1a5f9cc567f207bd50b4415 into b33bafe0044c48dabe3acce857aae963(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:53:56,054 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:56,054 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/B, priority=12, startTime=1680922435959; duration=0sec 2023-04-08 02:53:56,054 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:53:56,054 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:53:56,057 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49528 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:53:56,057 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/C is initiating minor compaction (all files) 2023-04-08 02:53:56,057 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/C in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:56,057 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/a5355e7e85a64193aac1abcb12dc1b0c, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/2c4fa4b2355740fc9456df420f6897f1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/a6147ebfa0594798b6e649b570fd6947, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/35d50f4578d34430a8bab026cd78fbd2] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=48.4 K 2023-04-08 02:53:56,058 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/A of 9a2d8d09b1a5f9cc567f207bd50b4415 into 0f6bacf9c35143d5b8dcdb050ee6adc6(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:53:56,058 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:56,058 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/A, priority=12, startTime=1680922435959; duration=0sec 2023-04-08 02:53:56,058 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:53:56,058 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting a5355e7e85a64193aac1abcb12dc1b0c, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=578, earliestPutTs=1680922434306 2023-04-08 02:53:56,059 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 2c4fa4b2355740fc9456df420f6897f1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=616, earliestPutTs=1680922434423 2023-04-08 02:53:56,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:56,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:56,060 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:56,060 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:56,060 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:56,060 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:56,061 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,061 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting a6147ebfa0594798b6e649b570fd6947, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=633, earliestPutTs=1680922434577 2023-04-08 02:53:56,061 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,061 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 35d50f4578d34430a8bab026cd78fbd2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=652, earliestPutTs=1680922435747 2023-04-08 02:53:56,062 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:56,062 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:56,063 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,063 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,063 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:56,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:56,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,064 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:56,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:56,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:56,072 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:56,072 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:56,072 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:56,089 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#C#compaction#84 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:53:56,089 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:56,089 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:56,089 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:56,089 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:56,090 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:56,090 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:56,090 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:56,090 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:56,090 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,090 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:56,090 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:56,096 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:56,096 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,096 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:56,096 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:56,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:56,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:56,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:56,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37f08b6b[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:56,101 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:56,101 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,102 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,102 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:56,102 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:56,102 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:56,104 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,104 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:56,114 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:56,114 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:56,114 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:56,114 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:56,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:56,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:56,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:56,118 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:56,118 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:56,118 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:56,119 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:56,119 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:56,120 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:56,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:56,121 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:56,121 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,121 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:56,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,123 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:56,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,124 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/3a7b578c6f854610a634497e4b0e78b3 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/3a7b578c6f854610a634497e4b0e78b3 2023-04-08 02:53:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:56,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,127 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,127 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,127 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,127 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:56,127 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:56,127 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,128 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,128 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:56,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:56,128 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:56,128 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:56,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:56,128 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:56,128 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:56,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:56,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:56,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:56,138 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/C of 9a2d8d09b1a5f9cc567f207bd50b4415 into 3a7b578c6f854610a634497e4b0e78b3(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:53:56,138 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:56,138 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/C, priority=12, startTime=1680922435959; duration=0sec 2023-04-08 02:53:56,139 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:53:56,139 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:56,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922496135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,144 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:56,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922496138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:56,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922496139, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:56,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922496145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:56,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922496145, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,229 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:56,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:56,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:56,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:56,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:56,234 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:56,234 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:56,234 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:56,234 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:56,234 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,234 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,236 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,236 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,236 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:56,236 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:56,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:56,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:56,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:56,238 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:56,238 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:56,238 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:56,238 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:56,238 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:56,247 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:56,247 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:56,248 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:56,248 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:56,248 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:56,248 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:56,248 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:56,248 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:56,248 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,248 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:56,248 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:56,250 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,250 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:56,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:56,250 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,250 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:56,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:56,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:56,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:56,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b169a35[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:56,254 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:56,254 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,257 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,257 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:56,257 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:56,257 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:56,257 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,257 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:56,259 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,259 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:56,259 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:56,259 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:56,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:56,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:56,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:56,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:56,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:56,262 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:56,262 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:56,262 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:56,262 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:56,262 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:56,263 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,263 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,263 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:56,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:56,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:56,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:56,264 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:56,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,265 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:56,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,269 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,269 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:56,269 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:56,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:56,269 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:56,275 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,275 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,275 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:56,275 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:56,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:56,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:56,275 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,276 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,276 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:56,276 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:56,276 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:56,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:56,342 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:56,342 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922496341, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,346 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:56,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922496346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,347 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:56,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922496346, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:56,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922496349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,351 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:56,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922496351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,376 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:56,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:56,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:56,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:56,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:56,381 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:56,381 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:56,381 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:56,381 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:56,381 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,381 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,402 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:56,402 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:56,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:56,402 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,403 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:56,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:56,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,403 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:56,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:56,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:56,404 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:56,404 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:56,404 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:56,404 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:56,404 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:56,404 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:56,404 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:56,404 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:56,404 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:56,404 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:56,404 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:56,404 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,404 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:56,405 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:56,425 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,425 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:56,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:56,425 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,426 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:56,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:56,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:56,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:56,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:56,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36bb0a13[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:56,427 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:56,427 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,460 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=667 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/a13d734e06134627b9fa867d93e375e9 2023-04-08 02:53:56,478 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=667 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/dabca8847a864d4393025973d67cf5bd 2023-04-08 02:53:56,479 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,479 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:56,479 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:56,479 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:56,480 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,480 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:56,502 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,502 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:56,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:56,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:56,502 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:56,502 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:56,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:56,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:56,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:56,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:56,505 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:56,505 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:56,505 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:56,505 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:56,505 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:56,525 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,525 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:56,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,525 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:56,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:56,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:56,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:56,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:56,526 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:56,528 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:56,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,609 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,609 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:56,609 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,609 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:56,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:56,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:56,609 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:56,609 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,610 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,610 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,610 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,610 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:56,610 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:56,610 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:56,610 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:56,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:56,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:56,610 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:56,646 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:56,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922496644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,650 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:56,650 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922496650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,651 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:56,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922496650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,655 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:56,655 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922496652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,656 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:56,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922496653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,710 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:56,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:56,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:56,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:56,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:56,716 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:56,716 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:56,716 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:56,716 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:56,716 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,716 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,717 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:56,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:56,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:56,718 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,720 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,720 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:56,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:56,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:56,721 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:56,721 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:56,722 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:56,722 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:56,722 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:56,722 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:56,722 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:56,722 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:56,722 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:56,724 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:56,724 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:56,724 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:56,724 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:56,724 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,724 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:56,724 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:56,726 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:56,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:56,727 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:56,727 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,727 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:56,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:56,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:56,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:56,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d60de52[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:56,732 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:56,732 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,734 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,734 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:56,734 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:56,735 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:56,735 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,735 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:56,738 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:56,738 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:56,738 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:56,738 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:56,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:56,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:56,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:56,744 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:56,744 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:56,744 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:56,744 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:56,744 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:56,744 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:56,746 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,746 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:56,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,747 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:56,747 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:56,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:56,748 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:56,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,749 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:56,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,752 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:56,752 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:56,752 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:56,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:56,752 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,753 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,753 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,753 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,753 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,753 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,753 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:56,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:56,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:56,753 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:56,753 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:56,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:56,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:56,853 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:56,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:56,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:56,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:56,856 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:56,857 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:56,857 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:56,857 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:56,857 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:56,857 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,857 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,859 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:56,859 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:56,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:56,860 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,860 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:56,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:56,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,860 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:56,861 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:56,861 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:56,861 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:56,861 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:56,861 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:56,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:56,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:56,862 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:56,862 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:56,862 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:56,862 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:56,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:56,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:56,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:56,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:56,864 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,864 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:56,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:56,864 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,864 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:56,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:56,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:56,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:56,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e78ad46[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:56,866 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:56,866 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,867 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,867 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:56,867 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:56,867 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:56,867 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,867 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:56,868 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,868 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:56,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:56,868 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:56,869 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:56,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:56,871 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:56,871 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:56,871 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:56,871 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:56,871 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:56,871 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:56,874 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,874 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,874 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:56,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:56,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:56,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:56,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:56,875 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:56,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,876 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:56,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:56,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,878 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,878 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,878 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,879 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,879 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,879 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:56,879 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:56,879 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:56,879 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:56,879 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:56,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:56,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:56,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:56,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:56,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:56,879 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:56,879 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:56,880 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:56,897 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=667 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/f482297469d644dd8f790fe1480cd675 2023-04-08 02:53:56,906 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/a13d734e06134627b9fa867d93e375e9 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/a13d734e06134627b9fa867d93e375e9 2023-04-08 02:53:56,915 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/a13d734e06134627b9fa867d93e375e9, entries=250, sequenceid=667, filesize=16.7 K 2023-04-08 02:53:56,916 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/dabca8847a864d4393025973d67cf5bd as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/dabca8847a864d4393025973d67cf5bd 2023-04-08 02:53:56,923 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/dabca8847a864d4393025973d67cf5bd, entries=150, sequenceid=667, filesize=11.9 K 2023-04-08 02:53:56,926 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/f482297469d644dd8f790fe1480cd675 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/f482297469d644dd8f790fe1480cd675 2023-04-08 02:53:56,954 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/f482297469d644dd8f790fe1480cd675, entries=150, sequenceid=667, filesize=11.9 K 2023-04-08 02:53:56,955 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 984ms, sequenceid=667, compaction requested=false 2023-04-08 02:53:56,955 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:56,980 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:56,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:56,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:56,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:56,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:56,988 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:56,988 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:56,988 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:56,988 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:56,989 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,989 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,990 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:56,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:56,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:56,990 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,991 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:56,991 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:56,991 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:56,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:56,992 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:56,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:56,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:56,996 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:56,996 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:56,996 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:57,008 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:57,008 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:57,008 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:57,008 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:57,008 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-08 02:53:57,009 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:53:57,009 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:57,009 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:53:57,009 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:57,009 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:53:57,009 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:57,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:57,157 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:57,170 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:57,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922497166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:57,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:57,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922497166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:57,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:57,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922497167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:57,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:57,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922497167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:57,175 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:57,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922497169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:57,274 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:57,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:57,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922497272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:57,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922497272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:57,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:57,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922497272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:57,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:57,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922497272, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:57,277 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:57,278 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922497276, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:57,437 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=687 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/4f7fa2fdf403442a92bc498fe52f61cc 2023-04-08 02:53:57,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:57,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:57,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922497476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:57,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922497476, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:57,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:57,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 215 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922497477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:57,478 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:57,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922497477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:57,481 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:57,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922497479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:57,780 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:57,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922497779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:57,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:57,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922497780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:57,781 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:57,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922497780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:57,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:57,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922497781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:57,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:57,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922497782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:57,855 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=687 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/2df7062a8d4b4e9db7207c1bfe71997f 2023-04-08 02:53:58,283 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:58,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922498282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,286 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:58,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922498284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:58,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922498286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,288 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:58,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922498286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,290 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:58,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922498287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,324 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=687 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/c71661957d354b6e892986d9cb75851c 2023-04-08 02:53:58,330 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/4f7fa2fdf403442a92bc498fe52f61cc as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/4f7fa2fdf403442a92bc498fe52f61cc 2023-04-08 02:53:58,344 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/4f7fa2fdf403442a92bc498fe52f61cc, entries=150, sequenceid=687, filesize=11.9 K 2023-04-08 02:53:58,345 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/2df7062a8d4b4e9db7207c1bfe71997f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/2df7062a8d4b4e9db7207c1bfe71997f 2023-04-08 02:53:58,350 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/2df7062a8d4b4e9db7207c1bfe71997f, entries=150, sequenceid=687, filesize=11.9 K 2023-04-08 02:53:58,351 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/c71661957d354b6e892986d9cb75851c as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/c71661957d354b6e892986d9cb75851c 2023-04-08 02:53:58,360 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/c71661957d354b6e892986d9cb75851c, entries=150, sequenceid=687, filesize=11.9 K 2023-04-08 02:53:58,363 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 1355ms, sequenceid=687, compaction requested=true 2023-04-08 02:53:58,363 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:58,363 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:58,363 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:58,364 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:58,364 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,364 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:58,364 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:58,367 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,367 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:58,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:58,367 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,367 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:58,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:58,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:58,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:58,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:58,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:58,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7120458e[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:58,370 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:58,370 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,372 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,373 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:58,373 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:58,373 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:58,373 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,373 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:58,374 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:58,374 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,374 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:58,374 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:58,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:58,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:58,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:58,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:58,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:58,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:58,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:58,378 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:58,378 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:58,378 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:58,378 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:58,378 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:58,378 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:58,380 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,380 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:58,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:58,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,380 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:58,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:58,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:58,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,381 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:58,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:58,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:58,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:58,382 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:58,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,386 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,386 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:58,386 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:58,386 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:58,386 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,386 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:58,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:58,386 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:58,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:58,386 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:58,386 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,386 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,386 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:58,386 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:58,387 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:58,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:58,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:58,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:58,487 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:58,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:58,489 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:58,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:58,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:58,490 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:58,490 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:58,490 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:58,490 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:58,491 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,491 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:58,495 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:58,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:58,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:58,495 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:58,495 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,496 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:58,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:58,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:58,496 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:58,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:58,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:58,497 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:58,497 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:58,497 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:58,497 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:58,497 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:58,497 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:58,497 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:58,497 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-08 02:53:58,497 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:53:58,497 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:58,497 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:53:58,498 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:58,498 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:53:58,498 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:58,507 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=702 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/f673339f1deb49ce867e88c66b695cfc 2023-04-08 02:53:58,520 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=702 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/732cfc5a06074905b33cd4db6110a782 2023-04-08 02:53:58,539 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=702 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/4abc3ec196fb4dcdad7d251caa2c1996 2023-04-08 02:53:58,545 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/f673339f1deb49ce867e88c66b695cfc as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/f673339f1deb49ce867e88c66b695cfc 2023-04-08 02:53:58,553 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/f673339f1deb49ce867e88c66b695cfc, entries=150, sequenceid=702, filesize=11.9 K 2023-04-08 02:53:58,554 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/732cfc5a06074905b33cd4db6110a782 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/732cfc5a06074905b33cd4db6110a782 2023-04-08 02:53:58,560 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/732cfc5a06074905b33cd4db6110a782, entries=150, sequenceid=702, filesize=11.9 K 2023-04-08 02:53:58,561 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/4abc3ec196fb4dcdad7d251caa2c1996 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/4abc3ec196fb4dcdad7d251caa2c1996 2023-04-08 02:53:58,574 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/4abc3ec196fb4dcdad7d251caa2c1996, entries=150, sequenceid=702, filesize=11.9 K 2023-04-08 02:53:58,575 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 78ms, sequenceid=702, compaction requested=true 2023-04-08 02:53:58,575 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:58,575 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:58,575 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:58,575 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:58,576 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,576 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:58,576 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:58,577 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,578 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:58,578 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,578 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:58,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:58,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:58,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:58,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:58,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fa5322b[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:58,584 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:58,584 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,587 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,587 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,587 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:58,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,588 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:58,588 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:58,588 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:58,590 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,590 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:58,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:58,590 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:58,590 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:58,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:58,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:58,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:58,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:58,594 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:58,594 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:58,594 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:58,594 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:58,594 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:58,596 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,596 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:58,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:58,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:58,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,598 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:58,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:58,599 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,602 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:58,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:58,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:58,605 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:58,605 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:58,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:58,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:58,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:58,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:58,605 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:58,605 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:58,605 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:58,605 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:58,606 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:58,605 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,606 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,606 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,606 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,605 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:58,608 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:58,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:58,608 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:58,706 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:58,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:58,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:58,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:58,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:58,709 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:58,709 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:58,709 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:58,709 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:58,710 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,710 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:58,712 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,712 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:58,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:58,713 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,713 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:58,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:58,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:58,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:58,714 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:58,715 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:58,715 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:58,715 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:58,715 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:58,715 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:58,715 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:58,715 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:58,715 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:58,715 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:58,715 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:58,715 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:58,716 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:58,716 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:58,716 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,716 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:58,716 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:58,717 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,717 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:58,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:58,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:58,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,719 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,719 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:58,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:58,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:58,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@446cedc3[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:58,720 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:58,720 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,721 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,722 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:58,722 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:58,722 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:58,723 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:58,723 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:58,723 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:58,723 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,724 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:58,724 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:58,724 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:58,724 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:58,724 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:58,724 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:58,724 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:58,727 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,727 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:58,727 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:58,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:58,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:58,728 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:58,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:58,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:58,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:58,730 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:58,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:58,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,737 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:58,737 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:58,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:58,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:58,737 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,737 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:58,737 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:58,737 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:58,737 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:58,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:58,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:58,737 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:58,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:58,737 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,738 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,738 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,738 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,738 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:58,738 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:58,838 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:58,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:58,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:58,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:58,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:58,842 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:58,842 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:58,842 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:58,843 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:58,843 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,843 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:58,844 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:58,844 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:58,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:58,845 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:58,845 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:58,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:58,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:58,846 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:58,850 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:58,850 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:58,850 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:58,850 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:58,850 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:58,856 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:58,856 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:58,856 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:58,856 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:58,856 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:58,856 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:58,856 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:58,856 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:58,856 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,856 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:58,856 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:58,858 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,858 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:58,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:58,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:58,858 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,858 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:58,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:58,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:58,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:58,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:58,864 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f556a95[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:58,864 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:58,864 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,866 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,867 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:58,867 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:58,867 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:58,867 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,867 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:58,868 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:58,868 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:58,868 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:58,868 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:58,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:58,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:58,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:58,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:58,872 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:58,872 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:58,873 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:58,873 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:58,873 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:58,873 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:58,874 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,874 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:58,874 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,874 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:58,875 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:58,875 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:58,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:58,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:58,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:58,878 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:58,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:58,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,883 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,883 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:58,883 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:58,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:58,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:58,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:58,883 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:58,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:58,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:58,883 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:58,883 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:58,883 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:58,883 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,883 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:58,883 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:58,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:58,886 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:58,886 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:58,986 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:58,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:58,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:58,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:58,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:58,998 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:58,998 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:58,998 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:58,998 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:58,999 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:58,999 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,001 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:59,001 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:59,001 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,001 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:59,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:59,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,002 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:59,002 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:59,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:59,002 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:59,002 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:59,002 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:59,003 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:59,003 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:59,003 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:59,003 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:59,003 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:59,005 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,005 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:59,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:59,005 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,005 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:59,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:59,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:59,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c2a5ec4[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:59,007 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:59,007 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,011 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,011 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:59,011 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:59,011 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:59,011 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,011 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:59,013 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,013 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:59,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:59,013 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:59,014 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:59,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:59,016 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:59,016 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:59,016 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:59,016 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:59,016 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:59,016 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:59,018 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,018 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:59,018 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:59,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:59,018 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,020 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,023 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,023 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,023 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:59,023 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,023 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:59,023 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:59,023 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:59,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:59,023 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,023 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,023 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:59,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:59,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:59,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:59,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:59,024 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:59,024 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:59,124 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:59,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:59,143 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:59,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:59,144 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:59,144 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:59,144 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:59,144 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:59,144 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:59,145 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,145 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,146 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,146 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:59,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:59,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:59,146 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,146 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:59,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:59,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,147 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:59,147 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:59,147 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:59,148 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:59,148 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:59,148 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:59,148 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:59,148 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:59,148 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:59,148 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:59,148 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:59,148 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:59,148 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:59,148 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,148 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:59,148 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:59,150 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,150 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:59,151 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:59,151 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:59,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:59,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a280300[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:59,152 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:59,152 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,154 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,154 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:59,154 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:59,154 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:59,154 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,154 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:59,155 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:59,155 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,155 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:59,155 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:59,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:59,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:59,158 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:59,158 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:59,158 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:59,158 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:59,158 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:59,160 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,160 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:59,160 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:59,160 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:59,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,162 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:59,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,166 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,166 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,166 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,166 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:59,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:59,166 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:59,166 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:59,166 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,166 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,166 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:59,166 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:59,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:59,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:59,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:59,166 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,166 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:59,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:59,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:59,266 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:59,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:59,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:59,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:59,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:59,270 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:59,270 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:59,270 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:59,270 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:59,270 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,270 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,272 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:59,272 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:59,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:59,272 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,272 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:59,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,272 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:59,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,273 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:59,273 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:59,273 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:59,273 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:59,273 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:59,273 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:59,273 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:59,273 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:59,273 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:59,274 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:59,274 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:59,274 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:59,274 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:59,274 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:59,274 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,274 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:59,274 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:59,275 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,275 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:59,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:59,276 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:59,276 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:59,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:59,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:59,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:59,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@909f173[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:59,277 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:59,277 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,279 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,279 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:59,279 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:59,279 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:59,279 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,279 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:59,281 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,281 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:59,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:59,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:59,281 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:59,281 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:59,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:59,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:59,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:59,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:59,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:59,284 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:59,284 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:59,284 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:59,284 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:59,284 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:59,286 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,286 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:59,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:59,286 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:59,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:59,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:59,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:59,287 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:59,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:59,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,293 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,295 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,295 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,295 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,296 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,296 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,296 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:59,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:59,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:59,295 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:59,296 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:59,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:59,296 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,297 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,297 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:59,297 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:59,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:59,297 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-08 02:53:59,297 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:53:59,297 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:59,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:59,297 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:53:59,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:59,298 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:59,298 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:53:59,298 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:59,315 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:59,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922499311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1680922439302, max timestamp=1680922439314Num uniques 859; 2023-04-08 02:53:59,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-08 02:53:59,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1680922439302, max timestamp=1680922439314Num uniques 859; 2023-04-08 02:53:59,315 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-08 02:53:59,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1680922439302, max timestamp=1680922439314Num uniques 859; 2023-04-08 02:53:59,316 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-08 02:53:59,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:59,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922499321, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:59,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922499322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,325 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:59,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922499323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,327 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:59,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922499325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,397 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:59,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:59,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:59,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:59,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:59,400 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:59,400 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:59,400 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:59,400 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:59,401 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,401 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,402 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:59,402 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:59,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:59,403 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,403 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:59,403 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:59,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,403 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:59,404 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:59,404 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:59,404 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:59,404 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:59,404 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:59,405 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:59,405 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:59,405 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:59,405 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:59,405 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:59,405 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:59,405 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:59,405 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:59,405 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,405 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:59,405 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:59,408 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,408 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:59,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:59,408 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,408 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:59,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:59,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:59,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:59,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:59,410 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dcc3305[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:59,410 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:59,410 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,411 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,411 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,411 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:59,411 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:59,411 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:59,411 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,411 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:59,415 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:59,415 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,415 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,415 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:59,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:59,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:59,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:59,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:59,418 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:59,418 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:59,418 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:59,418 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:59,418 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:59,418 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:59,419 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,419 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,419 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:59,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:59,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:59,420 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:59,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922499416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,420 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:59,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:59,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,421 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:59,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,423 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,424 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,424 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,424 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:59,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:59,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:59,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:59,424 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:59,424 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:59,424 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:59,424 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,424 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:59,424 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,424 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:59,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:59,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:59,424 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,424 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:59,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922499426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:59,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 281 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922499426, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,430 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:59,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922499427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,432 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:59,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922499428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,524 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:59,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:59,526 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:59,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:59,528 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:59,528 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:59,528 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:59,528 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:59,528 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:59,528 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,528 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,531 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:59,531 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:59,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:59,531 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,531 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:59,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:59,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:59,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:59,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:59,533 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:59,533 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:59,533 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:59,533 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:59,533 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:59,533 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:59,533 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:59,534 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:59,534 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:59,534 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:59,534 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:59,534 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,534 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:59,534 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:59,535 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:59,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:59,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:59,536 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:59,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,537 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,537 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:59,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:59,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:59,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35e8806[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:59,538 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:59,538 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,540 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,540 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:59,540 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:59,540 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:59,541 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,541 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:59,543 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:59,543 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,543 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:59,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:59,543 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:59,546 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:59,546 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:59,546 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:59,546 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:59,546 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:59,546 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:59,553 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,553 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:59,553 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:59,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:59,554 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:59,555 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:59,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,558 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,558 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,558 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,558 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:59,558 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:59,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:59,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:59,558 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:59,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:59,558 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:59,558 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,558 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,558 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,558 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:59,558 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:59,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:59,558 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:53:59,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:59,624 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:59,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 233 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922499621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,634 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:59,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 283 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922499631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,634 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:59,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922499631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,635 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:59,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922499631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,635 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:59,636 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922499634, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,658 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:59,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:59,661 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:59,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:59,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:59,663 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:59,663 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:59,663 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:59,663 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:59,663 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,664 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,665 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,665 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:59,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:59,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:59,665 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,665 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:59,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:59,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,666 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:59,666 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:59,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:59,666 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:59,667 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:59,667 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:59,667 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:59,667 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:59,667 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:59,667 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:59,667 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:59,667 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:59,667 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:59,667 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:59,667 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,667 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:59,667 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:59,669 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,669 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:59,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:59,669 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,669 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:59,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:59,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:59,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:59,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:59,671 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6904ab6e[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:59,671 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:59,671 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,673 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,673 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:59,673 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:59,673 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:59,673 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,673 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:59,674 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:59,674 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:59,674 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,674 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:59,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:59,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:59,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:59,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:59,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:59,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:59,677 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:59,677 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:59,678 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:59,678 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:59,678 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:59,678 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:59,679 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,679 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:59,679 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:59,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:59,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:59,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:59,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:59,680 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:59,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,681 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:59,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,682 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,684 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:59,684 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:59,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:59,684 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,684 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:59,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:59,684 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:59,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:59,684 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,684 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,684 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,684 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,684 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,684 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:59,684 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:59,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:59,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:59,706 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=718 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/d29902ce88d14223b87388b5e46c0718 2023-04-08 02:53:59,725 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=718 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/749ff7dca1e8426eacbabcfc74c3aa8a 2023-04-08 02:53:59,752 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=718 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/5ef8c6ef672a4529a3070cb83360f9a9 2023-04-08 02:53:59,761 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/d29902ce88d14223b87388b5e46c0718 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/d29902ce88d14223b87388b5e46c0718 2023-04-08 02:53:59,769 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/d29902ce88d14223b87388b5e46c0718, entries=150, sequenceid=718, filesize=11.9 K 2023-04-08 02:53:59,770 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/749ff7dca1e8426eacbabcfc74c3aa8a as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/749ff7dca1e8426eacbabcfc74c3aa8a 2023-04-08 02:53:59,775 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/749ff7dca1e8426eacbabcfc74c3aa8a, entries=150, sequenceid=718, filesize=11.9 K 2023-04-08 02:53:59,777 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/5ef8c6ef672a4529a3070cb83360f9a9 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/5ef8c6ef672a4529a3070cb83360f9a9 2023-04-08 02:53:59,783 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/5ef8c6ef672a4529a3070cb83360f9a9, entries=150, sequenceid=718, filesize=11.9 K 2023-04-08 02:53:59,784 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 487ms, sequenceid=718, compaction requested=true 2023-04-08 02:53:59,784 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:59,785 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:53:59,785 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 02:53:59,785 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 02:53:59,785 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:53:59,785 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:53:59,785 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:53:59,786 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:53:59,787 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 66730 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 02:53:59,788 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/A is initiating minor compaction (all files) 2023-04-08 02:53:59,788 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61850 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 02:53:59,788 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/A in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:59,788 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/B is initiating minor compaction (all files) 2023-04-08 02:53:59,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:53:59,788 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/B in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:59,788 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/0f6bacf9c35143d5b8dcdb050ee6adc6, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/a13d734e06134627b9fa867d93e375e9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/4f7fa2fdf403442a92bc498fe52f61cc, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/f673339f1deb49ce867e88c66b695cfc, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/d29902ce88d14223b87388b5e46c0718] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=65.2 K 2023-04-08 02:53:59,788 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/b33bafe0044c48dabe3acce857aae963, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/dabca8847a864d4393025973d67cf5bd, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/2df7062a8d4b4e9db7207c1bfe71997f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/732cfc5a06074905b33cd4db6110a782, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/749ff7dca1e8426eacbabcfc74c3aa8a] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=60.4 K 2023-04-08 02:53:59,789 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 0f6bacf9c35143d5b8dcdb050ee6adc6, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=652, earliestPutTs=1680922435747 2023-04-08 02:53:59,789 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting b33bafe0044c48dabe3acce857aae963, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=652, earliestPutTs=1680922435747 2023-04-08 02:53:59,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:53:59,789 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:53:59,789 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:53:59,789 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:59,790 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:53:59,790 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:53:59,790 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting a13d734e06134627b9fa867d93e375e9, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=667, earliestPutTs=1680922435858 2023-04-08 02:53:59,790 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,790 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,790 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting dabca8847a864d4393025973d67cf5bd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=667, earliestPutTs=1680922435858 2023-04-08 02:53:59,790 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 4f7fa2fdf403442a92bc498fe52f61cc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1680922435980 2023-04-08 02:53:59,791 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:59,791 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,792 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:59,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:59,792 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,792 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:53:59,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,792 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 2df7062a8d4b4e9db7207c1bfe71997f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1680922435980 2023-04-08 02:53:59,793 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,793 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting f673339f1deb49ce867e88c66b695cfc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1680922437162 2023-04-08 02:53:59,793 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 732cfc5a06074905b33cd4db6110a782, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1680922437162 2023-04-08 02:53:59,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:53:59,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,793 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:53:59,794 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting d29902ce88d14223b87388b5e46c0718, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=718, earliestPutTs=1680922439287 2023-04-08 02:53:59,794 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 749ff7dca1e8426eacbabcfc74c3aa8a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=718, earliestPutTs=1680922439287 2023-04-08 02:53:59,796 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:53:59,796 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:53:59,796 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:53:59,796 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:53:59,796 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:53:59,803 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:53:59,803 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:53:59,804 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:59,804 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:53:59,805 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=134.18 KB heapSize=356.86 KB 2023-04-08 02:53:59,805 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:53:59,805 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:53:59,806 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:53:59,806 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:53:59,807 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:53:59,807 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:53:59,814 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#B#compaction#96 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:53:59,821 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#A#compaction#97 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:53:59,849 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=749 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/184fbecf1b3a421eb4c22809f4e00863 2023-04-08 02:53:59,896 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=749 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/2be041c5e72b4397b2ff43e0f73bfbb7 2023-04-08 02:53:59,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:59,929 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:53:59,934 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=749 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/aa80eef02f854e19beb168b84fca968f 2023-04-08 02:53:59,940 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/184fbecf1b3a421eb4c22809f4e00863 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/184fbecf1b3a421eb4c22809f4e00863 2023-04-08 02:53:59,945 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/184fbecf1b3a421eb4c22809f4e00863, entries=150, sequenceid=749, filesize=11.9 K 2023-04-08 02:53:59,946 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/2be041c5e72b4397b2ff43e0f73bfbb7 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/2be041c5e72b4397b2ff43e0f73bfbb7 2023-04-08 02:53:59,951 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/2be041c5e72b4397b2ff43e0f73bfbb7, entries=150, sequenceid=749, filesize=11.9 K 2023-04-08 02:53:59,952 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/aa80eef02f854e19beb168b84fca968f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/aa80eef02f854e19beb168b84fca968f 2023-04-08 02:53:59,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:59,957 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922499944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,957 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:59,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922499946, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,958 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/aa80eef02f854e19beb168b84fca968f, entries=150, sequenceid=749, filesize=11.9 K 2023-04-08 02:53:59,958 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:53:59,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922499954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,959 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~356.81 KB/365376, currentSize=60.38 KB/61830 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 155ms, sequenceid=749, compaction requested=true 2023-04-08 02:53:59,959 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:53:59,959 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:53:59,959 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:53:59,959 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:53:59,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:53:59,959 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,959 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-08 02:53:59,960 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:53:59,960 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:53:59,960 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:53:59,960 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:59,960 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:53:59,960 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:59,960 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:53:59,960 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:53:59,961 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,961 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:59,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:59,962 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,962 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:53:59,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:59,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:59,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:53:59,964 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2fc05d54[Count = 0] remaining members to acquire global barrier 2023-04-08 02:53:59,964 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:53:59,964 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,967 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,967 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:53:59,967 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:53:59,967 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:53:59,967 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,967 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:53:59,969 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:53:59,969 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:59,969 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:53:59,969 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:59,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:59,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:59,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:53:59,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:53:59,975 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:53:59,975 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:53:59,975 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:53:59,975 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:53:59,975 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:53:59,977 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,977 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:59,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,978 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:53:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:53:59,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:53:59,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:53:59,979 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:53:59,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,981 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:53:59,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:53:59,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,986 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:53:59,986 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:53:59,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:53:59,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:53:59,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:53:59,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:53:59,987 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:53:59,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:53:59,987 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:53:59,988 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,988 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,988 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:53:59,988 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:53:59,988 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,988 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:53:59,988 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:53:59,988 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:53:59,988 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,000 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=762 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/b0139a9cd85744e7819e5877d2b5922d 2023-04-08 02:54:00,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:00,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922500008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,019 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:00,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 296 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922500010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,062 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:00,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922500058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,062 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:00,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922500059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,063 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:00,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922500059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,088 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:00,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:00,091 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:00,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:00,092 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:00,092 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:00,092 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:00,092 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:00,092 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:00,093 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,093 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:00,095 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:00,095 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:00,095 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,095 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:00,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:00,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:00,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:00,096 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:00,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:00,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:00,096 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:00,096 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:00,096 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:00,097 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:00,097 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:00,097 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:00,097 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:54:00,097 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:54:00,097 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:00,097 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:00,097 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:00,097 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,097 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:00,097 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:00,099 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,099 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:00,099 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:00,099 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:00,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:00,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:00,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:00,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33807b2[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:00,101 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:00,101 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:00,105 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:00,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:00,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:00,105 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:00,105 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:00,105 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:00,105 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,105 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:00,107 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,107 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:00,107 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:00,107 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:00,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:00,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:00,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:00,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:00,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:00,111 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:00,111 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:00,111 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:00,111 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:00,111 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:00,111 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:00,113 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,113 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:00,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,113 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:00,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:00,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:00,115 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:00,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:00,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,116 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:00,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:00,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,120 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,120 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:00,120 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:00,120 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,120 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:00,120 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:00,120 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:00,120 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:00,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:00,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:00,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:00,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:00,120 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,120 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:00,120 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:00,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:00,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:00,121 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:00,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 245 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922500120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,121 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:00,121 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:00,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 298 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922500120, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,221 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:00,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:00,224 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:00,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:00,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:00,231 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:00,231 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:00,231 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:00,231 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:00,231 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,231 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:00,241 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:00,241 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:00,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:00,242 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,242 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:00,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:00,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:00,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:00,244 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:00,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:00,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:00,244 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:00,244 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:00,244 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:00,250 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/86e2fdd4f57c48bfb7ae9f39ac81a068 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/86e2fdd4f57c48bfb7ae9f39ac81a068 2023-04-08 02:54:00,255 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:00,256 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:00,256 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:00,256 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:54:00,256 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:54:00,256 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:00,256 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:00,256 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:00,256 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,256 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:00,256 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:00,257 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/68c4f8d7e63649cbb4e39399a89aa711 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/68c4f8d7e63649cbb4e39399a89aa711 2023-04-08 02:54:00,257 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/B of 9a2d8d09b1a5f9cc567f207bd50b4415 into 86e2fdd4f57c48bfb7ae9f39ac81a068(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:00,258 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:54:00,258 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/B, priority=11, startTime=1680922439785; duration=0sec 2023-04-08 02:54:00,258 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:00,258 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-08 02:54:00,261 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:00,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:00,261 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:00,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:00,264 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 74036 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-08 02:54:00,264 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/C is initiating minor compaction (all files) 2023-04-08 02:54:00,264 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/C in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:00,265 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/3a7b578c6f854610a634497e4b0e78b3, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/f482297469d644dd8f790fe1480cd675, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/c71661957d354b6e892986d9cb75851c, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/4abc3ec196fb4dcdad7d251caa2c1996, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/5ef8c6ef672a4529a3070cb83360f9a9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/aa80eef02f854e19beb168b84fca968f] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=72.3 K 2023-04-08 02:54:00,265 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 3a7b578c6f854610a634497e4b0e78b3, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=652, earliestPutTs=1680922435747 2023-04-08 02:54:00,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:00,266 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:00,266 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:00,266 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting f482297469d644dd8f790fe1480cd675, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=667, earliestPutTs=1680922435858 2023-04-08 02:54:00,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:00,266 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting c71661957d354b6e892986d9cb75851c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=687, earliestPutTs=1680922435980 2023-04-08 02:54:00,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,266 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 4abc3ec196fb4dcdad7d251caa2c1996, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=702, earliestPutTs=1680922437162 2023-04-08 02:54:00,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:00,267 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 5ef8c6ef672a4529a3070cb83360f9a9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=718, earliestPutTs=1680922439287 2023-04-08 02:54:00,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:00,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7451a16b[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:00,267 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:00,267 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:00,267 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting aa80eef02f854e19beb168b84fca968f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=749, earliestPutTs=1680922439319 2023-04-08 02:54:00,268 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:00,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:00,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:00,269 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:00,269 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:00,269 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:00,269 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,269 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:00,269 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:00,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922500265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:00,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922500265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,270 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:00,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922500265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,271 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:00,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:00,272 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:00,272 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:00,272 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:00,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:00,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,275 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/A of 9a2d8d09b1a5f9cc567f207bd50b4415 into 68c4f8d7e63649cbb4e39399a89aa711(size=13.0 K), total size for store is 24.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:00,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:00,275 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:54:00,275 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/A, priority=11, startTime=1680922439785; duration=0sec 2023-04-08 02:54:00,275 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:00,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:00,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:00,277 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:00,277 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:00,277 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:00,277 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:00,277 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:00,279 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,279 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:00,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,279 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:00,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:00,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:00,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:00,280 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:00,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:00,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:00,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:00,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,282 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:00,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:00,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,286 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:00,286 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:00,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:00,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:00,286 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,286 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:00,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:00,286 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:00,286 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:00,287 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:00,287 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,287 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:00,287 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:00,287 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,287 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:00,287 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:00,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:00,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:00,287 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:00,294 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#C#compaction#103 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:00,310 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/ab0cb190cf824b72996ffdf36eeff776 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/ab0cb190cf824b72996ffdf36eeff776 2023-04-08 02:54:00,315 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/C of 9a2d8d09b1a5f9cc567f207bd50b4415 into ab0cb190cf824b72996ffdf36eeff776(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:00,315 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:54:00,315 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/C, priority=10, startTime=1680922439785; duration=0sec 2023-04-08 02:54:00,316 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:00,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:00,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922500322, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,324 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:00,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922500323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,387 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:00,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:00,389 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:00,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:00,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:00,390 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:00,391 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:00,391 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:00,391 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:00,391 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,391 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:00,392 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:00,392 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:00,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:00,393 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,393 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:00,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:00,394 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:00,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:00,394 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:00,397 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:00,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:00,397 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:00,397 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:00,397 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:00,397 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:00,397 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:00,397 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:00,397 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:54:00,397 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:54:00,397 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:00,397 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:00,397 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:00,398 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,398 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:00,398 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:00,400 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,400 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:00,400 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:00,400 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:00,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:00,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:00,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:00,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:00,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:00,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@38c22b54[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:00,402 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:00,402 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:00,403 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:00,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:00,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:00,403 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:00,403 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:00,403 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:00,403 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,403 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:00,404 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:00,404 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:00,404 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:00,404 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:00,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:00,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:00,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:00,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:00,407 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:00,407 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:00,408 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:00,408 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:00,408 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:00,408 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:00,409 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,409 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:00,409 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:00,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:00,410 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:00,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:00,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:00,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:00,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:00,411 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:00,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,414 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=762 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/a9c1ce441957472b8ed15e63031a4398 2023-04-08 02:54:00,416 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:00,416 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:00,416 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,417 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:00,417 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:00,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:00,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:00,417 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:00,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:00,417 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,417 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:00,417 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:00,417 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,417 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:00,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:00,417 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:00,418 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:00,418 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:00,418 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:00,429 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=762 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/9bacd3b806fb4d24b13833280353c163 2023-04-08 02:54:00,434 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/b0139a9cd85744e7819e5877d2b5922d as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/b0139a9cd85744e7819e5877d2b5922d 2023-04-08 02:54:00,438 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/b0139a9cd85744e7819e5877d2b5922d, entries=250, sequenceid=762, filesize=16.7 K 2023-04-08 02:54:00,440 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/a9c1ce441957472b8ed15e63031a4398 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/a9c1ce441957472b8ed15e63031a4398 2023-04-08 02:54:00,445 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/a9c1ce441957472b8ed15e63031a4398, entries=150, sequenceid=762, filesize=11.9 K 2023-04-08 02:54:00,446 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/9bacd3b806fb4d24b13833280353c163 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/9bacd3b806fb4d24b13833280353c163 2023-04-08 02:54:00,454 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/9bacd3b806fb4d24b13833280353c163, entries=150, sequenceid=762, filesize=11.9 K 2023-04-08 02:54:00,455 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 496ms, sequenceid=762, compaction requested=true 2023-04-08 02:54:00,455 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:54:00,455 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:00,455 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:54:00,455 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:00,455 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:00,455 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:54:00,457 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42528 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:54:00,457 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/A is initiating minor compaction (all files) 2023-04-08 02:54:00,457 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37648 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:54:00,457 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/A in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:00,457 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/B is initiating minor compaction (all files) 2023-04-08 02:54:00,457 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/B in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:00,457 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/68c4f8d7e63649cbb4e39399a89aa711, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/184fbecf1b3a421eb4c22809f4e00863, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/b0139a9cd85744e7819e5877d2b5922d] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=41.5 K 2023-04-08 02:54:00,457 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/86e2fdd4f57c48bfb7ae9f39ac81a068, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/2be041c5e72b4397b2ff43e0f73bfbb7, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/a9c1ce441957472b8ed15e63031a4398] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=36.8 K 2023-04-08 02:54:00,463 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 68c4f8d7e63649cbb4e39399a89aa711, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=718, earliestPutTs=1680922439287 2023-04-08 02:54:00,463 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 86e2fdd4f57c48bfb7ae9f39ac81a068, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=718, earliestPutTs=1680922439287 2023-04-08 02:54:00,464 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 2be041c5e72b4397b2ff43e0f73bfbb7, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=749, earliestPutTs=1680922439319 2023-04-08 02:54:00,464 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 184fbecf1b3a421eb4c22809f4e00863, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=749, earliestPutTs=1680922439319 2023-04-08 02:54:00,464 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting a9c1ce441957472b8ed15e63031a4398, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=762, earliestPutTs=1680922439953 2023-04-08 02:54:00,465 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting b0139a9cd85744e7819e5877d2b5922d, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=762, earliestPutTs=1680922439944 2023-04-08 02:54:00,478 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#B#compaction#105 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:00,481 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#A#compaction#106 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:00,505 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/6d74bc5b6cf9445da93626fb72573213 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/6d74bc5b6cf9445da93626fb72573213 2023-04-08 02:54:00,505 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/23266f53b1a446449809fb0860947dda as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/23266f53b1a446449809fb0860947dda 2023-04-08 02:54:00,512 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/A of 9a2d8d09b1a5f9cc567f207bd50b4415 into 23266f53b1a446449809fb0860947dda(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:00,512 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:54:00,512 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/A, priority=13, startTime=1680922440455; duration=0sec 2023-04-08 02:54:00,512 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:00,512 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-08 02:54:00,513 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-08 02:54:00,513 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-08 02:54:00,513 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. because compaction request was cancelled 2023-04-08 02:54:00,514 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/B of 9a2d8d09b1a5f9cc567f207bd50b4415 into 6d74bc5b6cf9445da93626fb72573213(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:00,514 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:54:00,514 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/B, priority=13, startTime=1680922440455; duration=0sec 2023-04-08 02:54:00,514 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:00,518 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:00,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:00,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:00,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:00,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:00,521 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:00,521 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:00,521 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:00,521 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:00,522 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,522 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:00,523 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:00,523 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:00,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:00,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:00,523 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,523 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:00,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:00,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:00,524 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:00,524 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:00,524 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:00,524 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:00,524 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:00,524 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:00,528 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:00,528 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:00,528 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:00,528 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:54:00,528 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-08 02:54:00,528 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:54:00,528 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:00,529 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:54:00,529 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:00,529 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:54:00,529 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:00,551 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=786 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/2433dc1c9dff4bf3833cd8ad71b89481 2023-04-08 02:54:00,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:54:00,575 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:54:00,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:00,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922500583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:00,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922500586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,588 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:00,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922500587, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,627 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:00,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922500626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,627 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:00,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922500627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,687 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:00,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922500687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,690 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:00,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922500689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,700 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:00,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922500697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,890 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:00,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922500889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,892 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:00,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922500892, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,902 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:00,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922500901, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:00,986 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=786 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/1faf3da9d5c9401b810abf8fb7657d02 2023-04-08 02:54:01,001 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=786 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/b10fcaa5e9f84ef4bac1001fb91f76e2 2023-04-08 02:54:01,007 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/2433dc1c9dff4bf3833cd8ad71b89481 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/2433dc1c9dff4bf3833cd8ad71b89481 2023-04-08 02:54:01,012 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/2433dc1c9dff4bf3833cd8ad71b89481, entries=150, sequenceid=786, filesize=11.9 K 2023-04-08 02:54:01,013 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/1faf3da9d5c9401b810abf8fb7657d02 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/1faf3da9d5c9401b810abf8fb7657d02 2023-04-08 02:54:01,020 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/1faf3da9d5c9401b810abf8fb7657d02, entries=150, sequenceid=786, filesize=11.9 K 2023-04-08 02:54:01,021 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/b10fcaa5e9f84ef4bac1001fb91f76e2 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/b10fcaa5e9f84ef4bac1001fb91f76e2 2023-04-08 02:54:01,025 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/b10fcaa5e9f84ef4bac1001fb91f76e2, entries=150, sequenceid=786, filesize=11.9 K 2023-04-08 02:54:01,029 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=73.80 KB/75570 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 501ms, sequenceid=786, compaction requested=true 2023-04-08 02:54:01,029 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:54:01,029 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:01,029 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:01,029 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:01,029 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,029 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:01,029 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:01,033 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,033 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,033 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:01,034 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:01,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:01,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:01,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:01,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:01,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1545ebe5[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:01,037 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:01,037 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,040 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,040 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,040 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:01,040 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:01,040 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:01,040 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:01,042 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:01,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:01,043 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:01,043 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:01,043 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:01,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:01,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:01,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:01,047 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:01,047 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:01,047 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:01,047 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:01,047 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:01,049 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,049 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,049 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:01,049 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:01,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:01,050 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:01,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:01,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,051 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:01,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,054 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:01,054 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:01,054 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,054 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,054 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,054 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:01,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:01,054 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:01,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:01,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:01,054 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,054 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,054 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,054 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:01,054 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:01,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:01,055 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:01,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:01,055 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:01,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:54:01,133 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-08 02:54:01,133 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:54:01,133 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:01,133 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:54:01,133 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:01,133 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:54:01,133 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:01,155 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:01,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:01,157 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:01,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:01,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:01,159 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:01,159 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:01,159 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:01,159 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:01,159 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,159 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,161 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:01,161 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922501158, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,161 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:01,161 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:01,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:01,162 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,162 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:01,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,162 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:01,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,163 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:01,163 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:01,163 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:01,163 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:01,163 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:01,163 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:01,164 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:01,164 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:01,164 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:01,164 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:54:01,164 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:54:01,164 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:01,164 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:01,164 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:01,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 312 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922501161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,165 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:01,165 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,165 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:01,165 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:01,166 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,166 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:01,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:01,167 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,167 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:01,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:01,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:01,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:01,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:01,169 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56837b68[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:01,169 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:01,169 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,170 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,170 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:01,170 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,170 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:01,170 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:01,170 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:01,172 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:01,172 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:01,172 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,173 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:01,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:01,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:01,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:01,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:01,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:01,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:01,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:01,175 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:01,175 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:01,175 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:01,175 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:01,175 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:01,177 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,177 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:01,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:01,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,177 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:01,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:01,178 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:01,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:01,178 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:01,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,179 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:01,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,182 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:01,182 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:01,182 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:01,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:01,182 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:01,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:01,182 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:01,182 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,182 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,182 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,182 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,182 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,182 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,182 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:01,182 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:01,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:01,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:01,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:01,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922501193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,194 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:01,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922501194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:01,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922501204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,263 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:01,264 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922501262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,266 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:01,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 314 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922501266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,283 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:01,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:01,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:01,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:01,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:01,286 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:01,286 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:01,286 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:01,286 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:01,287 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,287 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,288 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,288 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,288 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:01,288 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:01,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:01,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:01,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,289 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:01,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,289 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:01,290 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:01,290 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:01,290 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:01,290 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:01,290 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:01,290 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:01,290 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:01,290 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:01,290 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:54:01,290 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:54:01,290 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:01,290 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:01,290 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:01,290 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,290 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:01,290 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:01,293 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,293 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:01,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:01,293 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,293 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:01,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:01,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:01,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:01,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:01,295 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f20c22a[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:01,295 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:01,295 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,296 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,296 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:01,296 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,296 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:01,296 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:01,296 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:01,297 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,297 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:01,297 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:01,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:01,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:01,297 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:01,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:01,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:01,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:01,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:01,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:01,300 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:01,300 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:01,300 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:01,300 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:01,300 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:01,301 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,301 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,301 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,301 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:01,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:01,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:01,303 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:01,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:01,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:01,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,304 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:01,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,307 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,307 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,307 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:01,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:01,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:01,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:01,308 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:01,308 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:01,308 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:01,308 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,308 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,308 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,308 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,308 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:01,308 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:01,308 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:01,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:01,309 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:01,408 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:01,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:01,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:01,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:01,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:01,412 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:01,412 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:01,412 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:01,412 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:01,413 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,413 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,416 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,416 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:01,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:01,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:01,416 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,417 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:01,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,417 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:01,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,417 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:01,418 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:01,418 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:01,418 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:01,418 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:01,418 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:01,419 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:01,419 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:01,419 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:01,419 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:54:01,419 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:54:01,419 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:01,419 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:01,419 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:01,419 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,419 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:01,419 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:01,421 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,421 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:01,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:01,421 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,421 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:01,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:01,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:01,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:01,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:01,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@611d411[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:01,425 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:01,425 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,426 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,426 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:01,426 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:01,426 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:01,426 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,427 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:01,428 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,428 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:01,429 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:01,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:01,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:01,429 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:01,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:01,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:01,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:01,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:01,431 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:01,431 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:01,431 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:01,431 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:01,431 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:01,431 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:01,433 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,433 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,433 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:01,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:01,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:01,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:01,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:01,433 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:01,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:01,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,436 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,438 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,438 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,438 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,438 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:01,438 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:01,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:01,439 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:01,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:01,439 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,439 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,439 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,439 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:01,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:01,439 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:01,439 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:01,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:01,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:01,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 262 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922501466, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,467 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:01,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 316 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922501467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,540 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:01,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:01,542 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:01,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:01,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:01,544 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:01,544 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:01,544 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:01,544 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:01,544 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,544 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,546 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:01,546 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:01,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:01,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,546 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,546 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:01,546 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:01,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,547 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:01,547 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:01,547 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:01,547 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:01,547 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:01,547 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:01,549 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:01,549 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:01,549 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:01,549 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:54:01,549 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=802 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/3516c3443ed14d3cb00a60a0da88cdab 2023-04-08 02:54:01,549 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:54:01,549 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:01,549 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:01,549 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:01,549 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,549 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:01,549 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:01,552 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,552 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:01,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:01,552 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,552 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:01,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:01,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:01,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:01,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:01,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c311df5[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:01,554 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:01,554 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,555 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,555 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,555 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:01,555 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:01,555 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:01,555 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,555 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:01,556 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:01,557 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:01,556 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,557 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:01,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:01,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:01,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:01,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:01,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:01,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:01,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:01,560 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:01,560 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:01,560 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:01,560 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:01,560 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:01,563 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,563 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,563 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:01,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:01,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:01,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:01,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:01,563 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:01,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:01,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,564 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:01,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,568 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,568 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:01,568 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:01,568 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,568 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:01,568 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:01,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:01,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:01,568 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:01,568 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,568 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,568 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,568 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:01,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:01,568 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:01,568 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:01,569 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:01,669 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:01,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:01,671 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:01,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:01,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:01,672 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:01,672 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:01,672 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:01,672 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:01,673 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,673 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,674 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:01,674 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:01,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:01,674 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,674 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:01,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:01,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:01,675 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:01,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:01,676 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:01,676 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:01,676 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:01,676 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:01,676 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:01,676 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:01,676 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:54:01,676 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:54:01,676 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:01,676 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:01,676 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:01,676 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,676 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:01,676 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:01,678 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,678 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:01,678 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,678 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:01,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:01,680 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@71a46920[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:01,680 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:01,680 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,681 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,681 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,681 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:01,681 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:01,681 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:01,681 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:01,683 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,683 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:01,683 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:01,683 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:01,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:01,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:01,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:01,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:01,686 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:01,686 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:01,686 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:01,686 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:01,686 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:01,686 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:01,687 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,687 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:01,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:01,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,687 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:01,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:01,688 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:01,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:01,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:01,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,689 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:01,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,691 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,691 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,691 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,691 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:01,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:01,691 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:01,691 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:01,691 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:01,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:01,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:01,691 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:01,691 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:01,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:01,691 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:01,692 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,692 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:01,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:01,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 255 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922501696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922501696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:01,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922501711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,770 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:01,770 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 264 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922501769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,772 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:01,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922501772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,791 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:01,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:01,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:01,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:01,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:01,796 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:01,796 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:01,796 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:01,796 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:01,796 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,796 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,798 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:01,798 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:01,798 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,798 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:01,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,798 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:01,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,799 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:01,799 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:01,799 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:01,799 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:01,799 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:01,799 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:01,804 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:01,804 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:01,804 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:01,804 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:54:01,804 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:54:01,804 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:01,804 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:01,804 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:01,804 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,804 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:01,804 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:01,808 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:01,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:01,809 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,809 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:01,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:01,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:01,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:01,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:01,811 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@58bcc4ac[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:01,811 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:01,811 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,816 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,816 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:01,816 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:01,816 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:01,817 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,817 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:01,818 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:01,818 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,818 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:01,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:01,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:01,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:01,818 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:01,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:01,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:01,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:01,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:01,822 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:01,822 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:01,822 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:01,822 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:01,822 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:01,824 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,824 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:01,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,824 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:01,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:01,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:01,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:01,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,825 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:01,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:01,827 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:01,828 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,830 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,830 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,830 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,830 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,830 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,830 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,830 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:01,830 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:01,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:01,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:01,830 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:01,830 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,830 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:01,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:01,830 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:01,831 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:01,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:01,831 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:01,930 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:01,931 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:01,933 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:01,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:01,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:01,935 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:01,935 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:01,935 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:01,935 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:01,935 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,935 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,936 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:01,937 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,937 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:01,937 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,937 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:01,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,937 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:01,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,938 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:01,938 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:01,938 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:01,938 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:01,938 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:01,938 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:01,952 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:01,952 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:01,952 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:01,952 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:54:01,952 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:54:01,952 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:01,952 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:01,952 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:01,952 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,952 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:01,952 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:01,954 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,954 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:01,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:01,954 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,955 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:01,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:01,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:01,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:01,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:01,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@500522f6[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:01,956 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:01,956 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,957 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,958 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:01,958 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:01,958 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,958 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:01,958 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:01,961 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,961 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:01,961 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:01,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:01,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:01,962 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:01,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:01,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:01,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:01,964 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=802 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/422ec34edb5446619de7d2889e332c44 2023-04-08 02:54:01,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:01,965 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:01,965 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:01,965 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:01,965 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:01,965 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:01,965 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:01,968 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,968 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:01,968 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:01,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:01,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:01,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:01,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:01,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,969 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,970 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:01,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:01,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:01,975 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,975 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,975 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:01,975 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:01,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:01,975 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:01,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:01,975 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:01,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:01,976 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:01,976 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:01,975 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:01,976 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:01,976 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,976 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:01,976 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:01,976 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:01,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:01,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:01,991 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=802 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/c515adae0a8e43ee9bf6da2ebc545814 2023-04-08 02:54:01,996 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/3516c3443ed14d3cb00a60a0da88cdab as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/3516c3443ed14d3cb00a60a0da88cdab 2023-04-08 02:54:02,003 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/3516c3443ed14d3cb00a60a0da88cdab, entries=250, sequenceid=802, filesize=16.7 K 2023-04-08 02:54:02,012 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/422ec34edb5446619de7d2889e332c44 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/422ec34edb5446619de7d2889e332c44 2023-04-08 02:54:02,026 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/422ec34edb5446619de7d2889e332c44, entries=150, sequenceid=802, filesize=11.9 K 2023-04-08 02:54:02,027 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/c515adae0a8e43ee9bf6da2ebc545814 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/c515adae0a8e43ee9bf6da2ebc545814 2023-04-08 02:54:02,034 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/c515adae0a8e43ee9bf6da2ebc545814, entries=150, sequenceid=802, filesize=11.9 K 2023-04-08 02:54:02,035 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 902ms, sequenceid=802, compaction requested=true 2023-04-08 02:54:02,036 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:54:02,036 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:02,036 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:54:02,036 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:54:02,036 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:02,036 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:02,038 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37750 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:54:02,038 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/B is initiating minor compaction (all files) 2023-04-08 02:54:02,038 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/B in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:02,038 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/6d74bc5b6cf9445da93626fb72573213, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/1faf3da9d5c9401b810abf8fb7657d02, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/422ec34edb5446619de7d2889e332c44] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=36.9 K 2023-04-08 02:54:02,039 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 42630 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:54:02,039 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/A is initiating minor compaction (all files) 2023-04-08 02:54:02,039 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/A in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:02,039 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/23266f53b1a446449809fb0860947dda, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/2433dc1c9dff4bf3833cd8ad71b89481, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/3516c3443ed14d3cb00a60a0da88cdab] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=41.6 K 2023-04-08 02:54:02,039 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 6d74bc5b6cf9445da93626fb72573213, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=762, earliestPutTs=1680922439953 2023-04-08 02:54:02,040 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 23266f53b1a446449809fb0860947dda, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=762, earliestPutTs=1680922439953 2023-04-08 02:54:02,040 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 1faf3da9d5c9401b810abf8fb7657d02, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=786, earliestPutTs=1680922439997 2023-04-08 02:54:02,041 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 422ec34edb5446619de7d2889e332c44, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=802, earliestPutTs=1680922440584 2023-04-08 02:54:02,044 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 2433dc1c9dff4bf3833cd8ad71b89481, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=786, earliestPutTs=1680922439997 2023-04-08 02:54:02,044 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 3516c3443ed14d3cb00a60a0da88cdab, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=802, earliestPutTs=1680922440577 2023-04-08 02:54:02,068 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#A#compaction#113 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:02,076 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:02,077 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:02,077 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#B#compaction#114 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:02,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:02,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:02,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:02,104 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:02,104 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:02,104 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:02,104 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:02,105 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:02,105 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:02,107 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:02,107 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:02,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:02,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:02,108 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:02,108 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:02,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:02,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:02,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:02,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:02,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:02,124 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:02,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:02,124 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:02,124 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:02,124 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:02,129 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:02,129 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:02,129 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:02,130 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:54:02,130 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-08 02:54:02,130 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:54:02,130 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:02,130 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:54:02,130 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:02,130 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:54:02,130 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:02,153 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=818 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/fe66cc14c0464bde824062e94ecd0925 2023-04-08 02:54:02,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:54:02,276 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:54:02,300 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:02,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 273 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922502297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:02,301 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:02,301 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922502298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:02,402 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:02,402 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 275 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922502402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:02,403 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:02,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 328 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922502402, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:02,521 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/8a2dbcc82ab54ae9a29f9a8fed7af429 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/8a2dbcc82ab54ae9a29f9a8fed7af429 2023-04-08 02:54:02,526 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/A of 9a2d8d09b1a5f9cc567f207bd50b4415 into 8a2dbcc82ab54ae9a29f9a8fed7af429(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:02,526 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:54:02,526 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/A, priority=13, startTime=1680922442036; duration=0sec 2023-04-08 02:54:02,526 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:02,526 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:54:02,530 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49868 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:54:02,530 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/C is initiating minor compaction (all files) 2023-04-08 02:54:02,531 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/C in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:02,531 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/ab0cb190cf824b72996ffdf36eeff776, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/9bacd3b806fb4d24b13833280353c163, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/b10fcaa5e9f84ef4bac1001fb91f76e2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/c515adae0a8e43ee9bf6da2ebc545814] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=48.7 K 2023-04-08 02:54:02,531 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting ab0cb190cf824b72996ffdf36eeff776, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=749, earliestPutTs=1680922439319 2023-04-08 02:54:02,532 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 9bacd3b806fb4d24b13833280353c163, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=762, earliestPutTs=1680922439953 2023-04-08 02:54:02,532 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting b10fcaa5e9f84ef4bac1001fb91f76e2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=786, earliestPutTs=1680922439997 2023-04-08 02:54:02,533 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting c515adae0a8e43ee9bf6da2ebc545814, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=802, earliestPutTs=1680922440584 2023-04-08 02:54:02,537 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/be1b0891b7444161afac6d47e68c875d as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/be1b0891b7444161afac6d47e68c875d 2023-04-08 02:54:02,542 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#C#compaction#117 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:02,543 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/B of 9a2d8d09b1a5f9cc567f207bd50b4415 into be1b0891b7444161afac6d47e68c875d(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:02,543 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:54:02,543 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/B, priority=13, startTime=1680922442036; duration=0sec 2023-04-08 02:54:02,543 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:02,555 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/25e9af7fea7d4379831a609a7df98ee3 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/25e9af7fea7d4379831a609a7df98ee3 2023-04-08 02:54:02,559 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/C of 9a2d8d09b1a5f9cc567f207bd50b4415 into 25e9af7fea7d4379831a609a7df98ee3(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:02,559 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:54:02,559 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/C, priority=12, startTime=1680922442036; duration=0sec 2023-04-08 02:54:02,560 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:02,575 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=818 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/3c4f420fc9974bd08722ab2a7ef6b231 2023-04-08 02:54:02,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:02,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 277 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922502604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:02,605 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:02,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 330 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922502605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:02,703 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:02,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922502702, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:02,704 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:02,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 257 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922502704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:02,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:02,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922502713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:02,909 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:02,909 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:02,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 332 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922502908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:02,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 279 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922502908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:02,987 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=818 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/1d0d3d80b29949399d8384528dda7936 2023-04-08 02:54:02,995 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/fe66cc14c0464bde824062e94ecd0925 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/fe66cc14c0464bde824062e94ecd0925 2023-04-08 02:54:03,001 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/fe66cc14c0464bde824062e94ecd0925, entries=100, sequenceid=818, filesize=9.5 K 2023-04-08 02:54:03,003 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/3c4f420fc9974bd08722ab2a7ef6b231 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/3c4f420fc9974bd08722ab2a7ef6b231 2023-04-08 02:54:03,008 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/3c4f420fc9974bd08722ab2a7ef6b231, entries=100, sequenceid=818, filesize=9.5 K 2023-04-08 02:54:03,009 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/1d0d3d80b29949399d8384528dda7936 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/1d0d3d80b29949399d8384528dda7936 2023-04-08 02:54:03,014 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/1d0d3d80b29949399d8384528dda7936, entries=100, sequenceid=818, filesize=9.5 K 2023-04-08 02:54:03,015 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 885ms, sequenceid=818, compaction requested=false 2023-04-08 02:54:03,015 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:54:03,015 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:03,015 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:03,015 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:03,015 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:03,016 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:03,016 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:03,018 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:03,018 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:03,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:03,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:03,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:03,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:03,018 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:03,018 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:03,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:03,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:03,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:03,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:03,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:03,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64ef43b6[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:03,020 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:03,020 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:03,021 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:03,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:03,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:03,021 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:03,021 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:03,021 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:03,021 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:03,021 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:03,023 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:03,024 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:03,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:03,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:03,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:03,024 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:03,024 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:03,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:03,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:03,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:03,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:03,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:03,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:03,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:03,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:03,026 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:03,026 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:03,027 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:03,027 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:03,027 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:03,027 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:03,028 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:03,028 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:03,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:03,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:03,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:03,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:03,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:03,028 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:03,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:03,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:03,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:03,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:03,029 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:03,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:03,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:03,031 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:03,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:03,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:03,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:03,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:03,033 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:03,033 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:03,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:03,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:03,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:03,035 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:03,035 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:03,035 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:03,035 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:03,035 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:03,035 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:03,035 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:03,036 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:03,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:03,037 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:03,038 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:03,038 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:03,038 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:03,138 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:03,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:03,140 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:03,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:03,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:03,142 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:03,142 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:03,142 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:03,142 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:03,142 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:03,142 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:03,144 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:03,144 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:03,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:03,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:03,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:03,144 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:03,144 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:03,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:03,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:03,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:03,145 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:03,145 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:03,145 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:03,145 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:03,145 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:03,146 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:03,146 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:03,146 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:03,146 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:03,146 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:54:03,147 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-08 02:54:03,147 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:54:03,147 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:03,147 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:54:03,147 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:03,147 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:54:03,147 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:03,158 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=837 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/529eaa2966fc40a69adf1a3a0b997333 2023-04-08 02:54:03,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:54:03,412 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:54:03,435 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:03,435 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:03,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922503433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:03,435 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 287 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922503434, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:03,537 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:03,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 289 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922503536, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:03,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:03,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922503537, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:03,573 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=837 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/aa6e18bdc15b4625b0cc7435bd8e0f88 2023-04-08 02:54:03,740 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:03,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 291 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922503739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:03,741 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:03,741 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922503739, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:03,992 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=837 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/5ecd08f0204345d1aacfd96a2417a924 2023-04-08 02:54:03,998 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/529eaa2966fc40a69adf1a3a0b997333 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/529eaa2966fc40a69adf1a3a0b997333 2023-04-08 02:54:04,003 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/529eaa2966fc40a69adf1a3a0b997333, entries=150, sequenceid=837, filesize=11.9 K 2023-04-08 02:54:04,004 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/aa6e18bdc15b4625b0cc7435bd8e0f88 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/aa6e18bdc15b4625b0cc7435bd8e0f88 2023-04-08 02:54:04,009 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/aa6e18bdc15b4625b0cc7435bd8e0f88, entries=150, sequenceid=837, filesize=11.9 K 2023-04-08 02:54:04,010 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/5ecd08f0204345d1aacfd96a2417a924 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/5ecd08f0204345d1aacfd96a2417a924 2023-04-08 02:54:04,014 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/5ecd08f0204345d1aacfd96a2417a924, entries=150, sequenceid=837, filesize=11.9 K 2023-04-08 02:54:04,015 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 868ms, sequenceid=837, compaction requested=true 2023-04-08 02:54:04,015 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:54:04,015 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:04,015 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:04,015 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:04,016 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,016 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:04,016 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:04,017 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,017 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:04,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:04,017 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,017 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:04,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:04,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:04,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:04,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:04,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3650cb62[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:04,019 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:04,019 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,022 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,022 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:04,022 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:04,022 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:04,022 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,022 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:04,029 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:04,029 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:04,029 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,029 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:04,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:04,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:04,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:04,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:04,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:04,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:04,031 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:04,031 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:04,031 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:04,031 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:04,031 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:04,031 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:04,033 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,033 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,033 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:04,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:04,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:04,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:04,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:04,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:04,034 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:04,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,035 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:04,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,042 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:04,042 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:04,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:04,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:04,042 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:04,042 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:04,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:04,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:04,042 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,042 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,042 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,043 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,043 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,043 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,043 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:04,043 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:04,043 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:04,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:04,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:54:04,044 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-08 02:54:04,044 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:54:04,044 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:04,044 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:54:04,044 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:04,044 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:54:04,044 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:04,065 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:04,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 353 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922504063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:04,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 300 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922504064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,143 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:04,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:04,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:04,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:04,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:04,146 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:04,146 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:04,146 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:04,146 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:04,146 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,147 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,148 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:04,148 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,148 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:04,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:04,148 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,148 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:04,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,149 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:04,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,149 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:04,149 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:04,149 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:04,149 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:04,149 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:04,149 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:04,150 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:04,150 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:04,150 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:04,150 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:54:04,150 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:54:04,150 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:04,150 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:04,150 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:04,150 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,150 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:04,150 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:04,152 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,152 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:04,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:04,152 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,152 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:04,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:04,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:04,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:04,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:04,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@877f0a0[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:04,154 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:04,154 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,156 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,157 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:04,157 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:04,157 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:04,157 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,157 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:04,158 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,158 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:04,158 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:04,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:04,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:04,158 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:04,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:04,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:04,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:04,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:04,161 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:04,161 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:04,161 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:04,161 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:04,161 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:04,161 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:04,162 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,162 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:04,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:04,162 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:04,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:04,163 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,163 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:04,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:04,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:04,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,164 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:04,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,166 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,166 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:04,166 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:04,166 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,166 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:04,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:04,166 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,166 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,166 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,166 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:04,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:04,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:04,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:04,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:04,166 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:04,167 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:04,167 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:04,166 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,167 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,167 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:04,167 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:04,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 355 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922504167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 302 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922504167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,267 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:04,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:04,269 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:04,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:04,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:04,270 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:04,270 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:04,270 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:04,271 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:04,271 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,271 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,274 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,274 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:04,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:04,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:04,274 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,274 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:04,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:04,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,275 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:04,275 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:04,275 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:04,275 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:04,275 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:04,275 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:04,275 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:04,275 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:04,275 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:04,276 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:54:04,276 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:54:04,276 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:04,276 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:04,276 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:04,276 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,276 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:04,276 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:04,277 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,277 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:04,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:04,278 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,278 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:04,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:04,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:04,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:04,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:04,279 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:04,279 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@711c61ca[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:04,279 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,280 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,280 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,280 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:04,280 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:04,281 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:04,281 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,281 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:04,282 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,282 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:04,282 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:04,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:04,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:04,282 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:04,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:04,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:04,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:04,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:04,284 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:04,284 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:04,284 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:04,284 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:04,284 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:04,284 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:04,286 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,286 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:04,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,286 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:04,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:04,287 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:04,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:04,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,287 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:04,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:04,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,288 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:04,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,290 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,290 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,290 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,290 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,290 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,290 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,290 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,290 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:04,291 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:04,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:04,291 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:04,291 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:04,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:04,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:04,291 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:04,291 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:04,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:04,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:04,293 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:04,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:04,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 357 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922504368, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:04,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 304 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922504369, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,391 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:04,392 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:04,393 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:04,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:04,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:04,394 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:04,394 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:04,394 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:04,394 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:04,395 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,395 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,396 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:04,396 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:04,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:04,396 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,396 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:04,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,397 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:04,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,397 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:04,397 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:04,398 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:04,398 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:04,398 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:54:04,398 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:54:04,398 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:04,398 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:04,400 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,400 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:04,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:04,400 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,400 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:04,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:04,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:04,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:04,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:04,402 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ff07eac[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:04,402 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:04,402 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,408 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,408 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:04,408 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:04,408 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:04,408 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,408 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:04,410 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,410 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:04,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:04,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:04,410 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:04,410 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:04,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:04,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:04,413 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:04,413 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:04,413 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:04,413 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:04,413 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:04,413 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:04,414 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,414 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:04,414 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,415 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:04,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:04,415 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:04,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:04,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:04,415 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:04,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,419 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:04,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,425 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,425 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,425 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,425 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:04,425 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,425 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:04,426 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:04,426 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,426 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,425 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:04,426 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:04,426 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:04,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:04,426 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:04,426 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:04,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:04,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:04,462 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=854 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/d9a06eb1d6f640f3836a93c704aecdbf 2023-04-08 02:54:04,492 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=854 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/c09391f2298d41ff9085b3a33a158001 2023-04-08 02:54:04,526 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:04,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:04,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:04,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:04,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:04,529 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:04,529 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:04,529 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:04,529 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:04,530 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,530 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,531 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:04,531 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:04,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:04,531 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,531 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:04,532 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:04,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,532 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:04,532 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:04,532 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:04,532 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:04,532 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:04,532 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:04,533 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:04,533 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:04,533 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:04,533 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:54:04,533 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:54:04,533 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:04,533 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:04,533 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:04,533 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,533 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:04,533 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:04,534 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,535 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,535 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:04,535 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:04,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:04,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:04,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:04,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:04,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:04,537 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f4bb3eb[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:04,537 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:04,537 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,538 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,538 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:04,538 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,538 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:04,538 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:04,538 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:04,541 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,541 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:04,541 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:04,541 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:04,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:04,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:04,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:04,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:04,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:04,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:04,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:04,543 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:04,543 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:04,543 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:04,543 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:04,543 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:04,545 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,545 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,545 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:04,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:04,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:04,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:04,546 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:04,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,547 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:04,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,549 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:04,550 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:04,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:04,550 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,550 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,550 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,550 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,550 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:04,550 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:04,550 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,550 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:04,550 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,552 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:04,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:04,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:04,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:04,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:04,650 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:04,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:04,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:04,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:04,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:04,653 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:04,653 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:04,653 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:04,654 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:04,654 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,654 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,655 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:04,655 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:04,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:04,656 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,656 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:04,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:04,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,656 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:04,657 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:04,657 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:04,657 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:04,657 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:04,657 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:04,657 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:04,657 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:04,657 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:04,657 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:54:04,657 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:54:04,657 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:04,657 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:04,657 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:04,657 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,657 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:04,658 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:04,660 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,660 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:04,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:04,661 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,661 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:04,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:04,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:04,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:04,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:04,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@688eba52[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:04,662 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:04,662 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,664 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,664 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:04,664 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:04,664 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:04,664 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,664 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:04,665 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,665 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:04,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:04,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:04,665 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:04,666 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:04,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:04,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:04,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:04,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:04,668 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:04,668 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:04,668 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:04,668 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:04,668 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:04,668 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:04,669 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,669 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:04,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:04,670 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:04,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:04,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:04,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:04,671 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:04,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,672 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,672 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:04,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:04,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 306 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39442 deadline: 1680922504672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:04,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 359 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39438 deadline: 1680922504673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,676 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,676 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:04,676 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:04,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:04,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:04,676 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:04,676 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:04,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:04,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:04,676 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,676 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,676 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,676 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,676 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,676 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:04,676 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:04,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:04,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:04,711 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:04,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39498 deadline: 1680922504711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,713 DEBUG [Thread-141] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4125 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:54:04,717 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:04,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 259 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39458 deadline: 1680922504713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,719 DEBUG [Thread-145] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4136 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:54:04,721 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:04,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:39470 deadline: 1680922504720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,723 DEBUG [Thread-147] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4135 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:54:04,776 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:04,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:04,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:04,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:04,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:04,780 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:04,780 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:04,780 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:04,780 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:04,780 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,780 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,781 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,781 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:04,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:04,782 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,782 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:04,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,782 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:04,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,782 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:04,783 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:04,783 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:04,783 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:04,783 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:04,783 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:04,783 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:04,783 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:04,783 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:04,783 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. started... 2023-04-08 02:54:04,783 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. as already flushing 2023-04-08 02:54:04,783 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:04,783 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:04,784 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:04,784 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,784 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:04,784 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:04,788 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,788 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:04,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:04,788 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,788 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:04,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:04,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:04,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:04,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:04,790 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:04,790 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a8e94b8[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:04,790 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,791 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,792 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:04,792 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:04,792 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:04,792 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,792 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:04,797 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,797 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:04,797 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:04,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:04,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:04,797 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:04,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:04,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:04,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:04,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:04,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:04,799 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:04,799 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:04,799 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:04,799 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:04,800 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:04,801 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,801 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,801 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:04,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:04,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:04,801 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:04,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:04,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:04,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:04,803 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:04,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:04,807 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:04,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:04,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:04,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:04,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:04,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:04,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:04,808 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:04,808 DEBUG [Thread-151] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:04,808 DEBUG [Thread-151] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:04,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:04,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:04,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:04,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:04,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:04,809 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:04,893 DEBUG [Thread-154] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x59d6b6ea to 127.0.0.1:64826 2023-04-08 02:54:04,893 DEBUG [Thread-154] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:54:04,897 DEBUG [Thread-152] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x49d71120 to 127.0.0.1:64826 2023-04-08 02:54:04,897 DEBUG [Thread-152] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:54:04,898 DEBUG [Thread-156] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0d24bbb7 to 127.0.0.1:64826 2023-04-08 02:54:04,898 DEBUG [Thread-156] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:54:04,898 DEBUG [Thread-158] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x63871a9d to 127.0.0.1:64826 2023-04-08 02:54:04,898 DEBUG [Thread-158] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:54:04,908 DEBUG [Thread-151] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:04,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:04,920 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=854 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/0655708188504148b47f117b770e402d 2023-04-08 02:54:04,925 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/d9a06eb1d6f640f3836a93c704aecdbf as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/d9a06eb1d6f640f3836a93c704aecdbf 2023-04-08 02:54:04,929 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/d9a06eb1d6f640f3836a93c704aecdbf, entries=200, sequenceid=854, filesize=14.3 K 2023-04-08 02:54:04,930 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/c09391f2298d41ff9085b3a33a158001 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/c09391f2298d41ff9085b3a33a158001 2023-04-08 02:54:04,934 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/c09391f2298d41ff9085b3a33a158001, entries=150, sequenceid=854, filesize=11.9 K 2023-04-08 02:54:04,935 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/0655708188504148b47f117b770e402d as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/0655708188504148b47f117b770e402d 2023-04-08 02:54:04,940 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/0655708188504148b47f117b770e402d, entries=150, sequenceid=854, filesize=11.9 K 2023-04-08 02:54:04,940 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 896ms, sequenceid=854, compaction requested=true 2023-04-08 02:54:04,941 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:54:04,941 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:04,941 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:54:04,941 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:04,941 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:54:04,941 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:04,942 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 50034 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:54:04,942 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/A is initiating minor compaction (all files) 2023-04-08 02:54:04,942 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47594 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:54:04,942 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/A in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:04,942 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/B is initiating minor compaction (all files) 2023-04-08 02:54:04,943 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/8a2dbcc82ab54ae9a29f9a8fed7af429, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/fe66cc14c0464bde824062e94ecd0925, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/529eaa2966fc40a69adf1a3a0b997333, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/d9a06eb1d6f640f3836a93c704aecdbf] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=48.9 K 2023-04-08 02:54:04,943 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/B in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:04,943 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/be1b0891b7444161afac6d47e68c875d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/3c4f420fc9974bd08722ab2a7ef6b231, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/aa6e18bdc15b4625b0cc7435bd8e0f88, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/c09391f2298d41ff9085b3a33a158001] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=46.5 K 2023-04-08 02:54:04,943 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 8a2dbcc82ab54ae9a29f9a8fed7af429, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=802, earliestPutTs=1680922440584 2023-04-08 02:54:04,943 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting be1b0891b7444161afac6d47e68c875d, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=802, earliestPutTs=1680922440584 2023-04-08 02:54:04,943 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting fe66cc14c0464bde824062e94ecd0925, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=818, earliestPutTs=1680922441155 2023-04-08 02:54:04,944 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 3c4f420fc9974bd08722ab2a7ef6b231, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=818, earliestPutTs=1680922441155 2023-04-08 02:54:04,944 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 529eaa2966fc40a69adf1a3a0b997333, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=837, earliestPutTs=1680922442292 2023-04-08 02:54:04,944 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting aa6e18bdc15b4625b0cc7435bd8e0f88, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=837, earliestPutTs=1680922442292 2023-04-08 02:54:04,944 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting d9a06eb1d6f640f3836a93c704aecdbf, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=854, earliestPutTs=1680922443430 2023-04-08 02:54:04,944 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting c09391f2298d41ff9085b3a33a158001, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=854, earliestPutTs=1680922443432 2023-04-08 02:54:04,953 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#B#compaction#126 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:04,954 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#A#compaction#125 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:05,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:54:05,176 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-08 02:54:05,176 DEBUG [Thread-149] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02c57e1b to 127.0.0.1:64826 2023-04-08 02:54:05,176 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:54:05,176 DEBUG [Thread-149] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:54:05,176 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:05,177 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:54:05,177 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:05,177 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:54:05,177 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:05,181 DEBUG [Thread-143] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x56223d42 to 127.0.0.1:64826 2023-04-08 02:54:05,181 DEBUG [Thread-143] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:54:05,364 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/434c49ee61054dc78871454993f38627 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/434c49ee61054dc78871454993f38627 2023-04-08 02:54:05,364 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/0c653dd45e16493abd81bd0eb92ee579 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/0c653dd45e16493abd81bd0eb92ee579 2023-04-08 02:54:05,369 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/A of 9a2d8d09b1a5f9cc567f207bd50b4415 into 434c49ee61054dc78871454993f38627(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:05,369 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/B of 9a2d8d09b1a5f9cc567f207bd50b4415 into 0c653dd45e16493abd81bd0eb92ee579(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:05,369 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:54:05,369 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:54:05,369 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/A, priority=12, startTime=1680922444941; duration=0sec 2023-04-08 02:54:05,369 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/B, priority=12, startTime=1680922444941; duration=0sec 2023-04-08 02:54:05,369 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:05,369 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:05,369 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:54:05,371 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47560 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:54:05,371 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 9a2d8d09b1a5f9cc567f207bd50b4415/C is initiating minor compaction (all files) 2023-04-08 02:54:05,371 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 9a2d8d09b1a5f9cc567f207bd50b4415/C in TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:05,371 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/25e9af7fea7d4379831a609a7df98ee3, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/1d0d3d80b29949399d8384528dda7936, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/5ecd08f0204345d1aacfd96a2417a924, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/0655708188504148b47f117b770e402d] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp, totalSize=46.4 K 2023-04-08 02:54:05,371 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 25e9af7fea7d4379831a609a7df98ee3, keycount=150, bloomtype=ROW, size=13.1 K, encoding=NONE, compression=NONE, seqNum=802, earliestPutTs=1680922440584 2023-04-08 02:54:05,372 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 1d0d3d80b29949399d8384528dda7936, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=818, earliestPutTs=1680922441155 2023-04-08 02:54:05,372 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 5ecd08f0204345d1aacfd96a2417a924, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=837, earliestPutTs=1680922442292 2023-04-08 02:54:05,372 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 0655708188504148b47f117b770e402d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=854, earliestPutTs=1680922443432 2023-04-08 02:54:05,380 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 9a2d8d09b1a5f9cc567f207bd50b4415#C#compaction#128 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:05,585 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=870 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/188ef647a35142ce96f16f29d6ec20d7 2023-04-08 02:54:05,602 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=870 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/c6e4135280fc4c71bf5e910289e5517f 2023-04-08 02:54:05,789 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/236cdc0742f64fcd82006df3e370b148 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/236cdc0742f64fcd82006df3e370b148 2023-04-08 02:54:05,794 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 9a2d8d09b1a5f9cc567f207bd50b4415/C of 9a2d8d09b1a5f9cc567f207bd50b4415 into 236cdc0742f64fcd82006df3e370b148(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:05,794 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:54:05,794 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415., storeName=9a2d8d09b1a5f9cc567f207bd50b4415/C, priority=12, startTime=1680922444941; duration=0sec 2023-04-08 02:54:05,795 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:06,013 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=870 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/e7b505b094dc400bae3fb45391bf8d66 2023-04-08 02:54:06,018 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/188ef647a35142ce96f16f29d6ec20d7 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/188ef647a35142ce96f16f29d6ec20d7 2023-04-08 02:54:06,022 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/188ef647a35142ce96f16f29d6ec20d7, entries=150, sequenceid=870, filesize=11.9 K 2023-04-08 02:54:06,023 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/c6e4135280fc4c71bf5e910289e5517f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/c6e4135280fc4c71bf5e910289e5517f 2023-04-08 02:54:06,027 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/c6e4135280fc4c71bf5e910289e5517f, entries=150, sequenceid=870, filesize=11.9 K 2023-04-08 02:54:06,028 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/e7b505b094dc400bae3fb45391bf8d66 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/e7b505b094dc400bae3fb45391bf8d66 2023-04-08 02:54:06,031 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/e7b505b094dc400bae3fb45391bf8d66, entries=150, sequenceid=870, filesize=11.9 K 2023-04-08 02:54:06,032 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=6.71 KB/6870 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 856ms, sequenceid=870, compaction requested=false 2023-04-08 02:54:06,032 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:54:08,717 DEBUG [Thread-141] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x748f869b to 127.0.0.1:64826 2023-04-08 02:54:08,717 DEBUG [Thread-141] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:54:08,727 DEBUG [Thread-145] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x30eb7fb2 to 127.0.0.1:64826 2023-04-08 02:54:08,727 DEBUG [Thread-145] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:54:08,754 DEBUG [Thread-147] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x2de4d1f6 to 127.0.0.1:64826 2023-04-08 02:54:08,754 DEBUG [Thread-147] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:54:08,755 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-08 02:54:08,755 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 120 2023-04-08 02:54:08,755 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 211 2023-04-08 02:54:08,755 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 131 2023-04-08 02:54:08,755 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 106 2023-04-08 02:54:08,755 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 172 2023-04-08 02:54:08,755 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-08 02:54:08,755 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(398): read 4596 2023-04-08 02:54:08,755 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(398): read 4597 2023-04-08 02:54:08,755 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-08 02:54:08,755 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(402): scanned 2051 2023-04-08 02:54:08,755 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(403): verified 6149 rows 2023-04-08 02:54:08,755 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(402): scanned 2043 2023-04-08 02:54:08,755 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(403): verified 6128 rows 2023-04-08 02:54:08,755 INFO [Listener at localhost/39741] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-08 02:54:08,755 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08c3f619 to 127.0.0.1:64826 2023-04-08 02:54:08,755 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:54:08,761 INFO [Listener at localhost/39741] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-08 02:54:08,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-08 02:54:08,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure2.ProcedureExecutor(1029): Stored pid=12, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-08 02:54:08,776 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680922448775"}]},"ts":"1680922448775"} 2023-04-08 02:54:08,776 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-08 02:54:08,777 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-08 02:54:08,778 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-08 02:54:08,780 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9a2d8d09b1a5f9cc567f207bd50b4415, UNASSIGN}] 2023-04-08 02:54:08,781 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=13, ppid=12, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=9a2d8d09b1a5f9cc567f207bd50b4415, UNASSIGN 2023-04-08 02:54:08,782 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=9a2d8d09b1a5f9cc567f207bd50b4415, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:08,782 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922448782"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680922448782"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680922448782"}]},"ts":"1680922448782"} 2023-04-08 02:54:08,784 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=14, ppid=13, state=RUNNABLE; CloseRegionProcedure 9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983}] 2023-04-08 02:54:08,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-08 02:54:08,945 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:54:08,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 9a2d8d09b1a5f9cc567f207bd50b4415, disabling compactions & flushes 2023-04-08 02:54:08,945 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:08,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:08,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. after waiting 0 ms 2023-04-08 02:54:08,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:08,945 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 9a2d8d09b1a5f9cc567f207bd50b4415 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-08 02:54:08,945 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=A 2023-04-08 02:54:08,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:08,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=B 2023-04-08 02:54:08,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:08,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 9a2d8d09b1a5f9cc567f207bd50b4415, store=C 2023-04-08 02:54:08,946 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:09,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-08 02:54:09,356 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=880 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/8d97c1b954094d0f8ebbce47a1cfd753 2023-04-08 02:54:09,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-08 02:54:09,768 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=880 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/6eb1c0f7995640d6b5bff486da173f29 2023-04-08 02:54:09,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-08 02:54:10,180 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=880 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/529dc5f2b6464be4ae55bcfb004fbd32 2023-04-08 02:54:10,185 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/A/8d97c1b954094d0f8ebbce47a1cfd753 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/8d97c1b954094d0f8ebbce47a1cfd753 2023-04-08 02:54:10,189 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/8d97c1b954094d0f8ebbce47a1cfd753, entries=100, sequenceid=880, filesize=9.5 K 2023-04-08 02:54:10,190 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/B/6eb1c0f7995640d6b5bff486da173f29 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/6eb1c0f7995640d6b5bff486da173f29 2023-04-08 02:54:10,194 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/6eb1c0f7995640d6b5bff486da173f29, entries=100, sequenceid=880, filesize=9.5 K 2023-04-08 02:54:10,195 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/.tmp/C/529dc5f2b6464be4ae55bcfb004fbd32 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/529dc5f2b6464be4ae55bcfb004fbd32 2023-04-08 02:54:10,199 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/529dc5f2b6464be4ae55bcfb004fbd32, entries=100, sequenceid=880, filesize=9.5 K 2023-04-08 02:54:10,200 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for 9a2d8d09b1a5f9cc567f207bd50b4415 in 1255ms, sequenceid=880, compaction requested=true 2023-04-08 02:54:10,225 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/c496f25dff044743a66f1e06131fa6dc, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/3f59fb0598cd4e448ab979202798dbbc, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/d3180eb0a9094c04953acf896f907f20, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/b6f42db44256477082734e60146a733c, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/d39bdab3821d4346b3be7e6a8517746c, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/3cd2840b95084349a0dc712536e2e051, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/fa682d0f3f9f4885b2d32934b5046182, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/aaeb958413ee4a7eb33a80acf67feb63, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/dd588e641ae5448b876e467fa8703acd, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/ac381fac7b9549348a0ad441bc65cd43, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/72319d67dc5e40a0b8e2109413912deb, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/c025dc22230c4ddda694ca4e39267b4f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/45dbb59b772d49cca43b62e8a2d201c7, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/5faee44d217b4f0b83baf62a4713bc91, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/123a121f9639404f81c7b5dcad954138, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/41100221206d482091dfa1b977b233b8, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/9fd2e1f5a9944ad5851b16bfbf0d2c35, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/cd6570ec02584af483729deaa47f7ae4, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/06ca2fc9d4d64c0a8befbace17c70bd2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/de4043d0d9444a80ab5a6d0a20118ceb, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/a391d164dd7f4ceaac2017e6451bba5e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/9a7c7ffceca4474caf5a8816c62b1bef, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/e1d948bc142642d8a5b019b0f73cea95, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/e890df3a7eda4aeaa6ea5563e0d966c1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/93b0b7a4dac34519bc0900b5a8880c40, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/4abfff4b5b434458afab50099f879917, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/42b03fff9cf24ee6a6ba18ffbf0bcee9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/0f6bacf9c35143d5b8dcdb050ee6adc6, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/a13d734e06134627b9fa867d93e375e9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/4f7fa2fdf403442a92bc498fe52f61cc, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/f673339f1deb49ce867e88c66b695cfc, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/68c4f8d7e63649cbb4e39399a89aa711, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/d29902ce88d14223b87388b5e46c0718, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/184fbecf1b3a421eb4c22809f4e00863, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/b0139a9cd85744e7819e5877d2b5922d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/23266f53b1a446449809fb0860947dda, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/2433dc1c9dff4bf3833cd8ad71b89481, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/3516c3443ed14d3cb00a60a0da88cdab, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/8a2dbcc82ab54ae9a29f9a8fed7af429, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/fe66cc14c0464bde824062e94ecd0925, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/529eaa2966fc40a69adf1a3a0b997333, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/d9a06eb1d6f640f3836a93c704aecdbf] to archive 2023-04-08 02:54:10,226 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 02:54:10,232 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/c496f25dff044743a66f1e06131fa6dc to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/c496f25dff044743a66f1e06131fa6dc 2023-04-08 02:54:10,234 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/3f59fb0598cd4e448ab979202798dbbc to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/3f59fb0598cd4e448ab979202798dbbc 2023-04-08 02:54:10,237 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/d3180eb0a9094c04953acf896f907f20 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/d3180eb0a9094c04953acf896f907f20 2023-04-08 02:54:10,238 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/b6f42db44256477082734e60146a733c to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/b6f42db44256477082734e60146a733c 2023-04-08 02:54:10,240 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/d39bdab3821d4346b3be7e6a8517746c to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/d39bdab3821d4346b3be7e6a8517746c 2023-04-08 02:54:10,242 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/3cd2840b95084349a0dc712536e2e051 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/3cd2840b95084349a0dc712536e2e051 2023-04-08 02:54:10,243 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/fa682d0f3f9f4885b2d32934b5046182 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/fa682d0f3f9f4885b2d32934b5046182 2023-04-08 02:54:10,245 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/aaeb958413ee4a7eb33a80acf67feb63 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/aaeb958413ee4a7eb33a80acf67feb63 2023-04-08 02:54:10,246 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/dd588e641ae5448b876e467fa8703acd to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/dd588e641ae5448b876e467fa8703acd 2023-04-08 02:54:10,248 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/ac381fac7b9549348a0ad441bc65cd43 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/ac381fac7b9549348a0ad441bc65cd43 2023-04-08 02:54:10,249 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/72319d67dc5e40a0b8e2109413912deb to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/72319d67dc5e40a0b8e2109413912deb 2023-04-08 02:54:10,251 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/c025dc22230c4ddda694ca4e39267b4f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/c025dc22230c4ddda694ca4e39267b4f 2023-04-08 02:54:10,252 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/45dbb59b772d49cca43b62e8a2d201c7 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/45dbb59b772d49cca43b62e8a2d201c7 2023-04-08 02:54:10,253 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/5faee44d217b4f0b83baf62a4713bc91 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/5faee44d217b4f0b83baf62a4713bc91 2023-04-08 02:54:10,255 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/123a121f9639404f81c7b5dcad954138 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/123a121f9639404f81c7b5dcad954138 2023-04-08 02:54:10,256 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/41100221206d482091dfa1b977b233b8 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/41100221206d482091dfa1b977b233b8 2023-04-08 02:54:10,258 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/9fd2e1f5a9944ad5851b16bfbf0d2c35 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/9fd2e1f5a9944ad5851b16bfbf0d2c35 2023-04-08 02:54:10,259 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/cd6570ec02584af483729deaa47f7ae4 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/cd6570ec02584af483729deaa47f7ae4 2023-04-08 02:54:10,261 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/06ca2fc9d4d64c0a8befbace17c70bd2 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/06ca2fc9d4d64c0a8befbace17c70bd2 2023-04-08 02:54:10,263 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/de4043d0d9444a80ab5a6d0a20118ceb to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/de4043d0d9444a80ab5a6d0a20118ceb 2023-04-08 02:54:10,265 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/a391d164dd7f4ceaac2017e6451bba5e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/a391d164dd7f4ceaac2017e6451bba5e 2023-04-08 02:54:10,266 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/9a7c7ffceca4474caf5a8816c62b1bef to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/9a7c7ffceca4474caf5a8816c62b1bef 2023-04-08 02:54:10,268 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/e1d948bc142642d8a5b019b0f73cea95 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/e1d948bc142642d8a5b019b0f73cea95 2023-04-08 02:54:10,269 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/e890df3a7eda4aeaa6ea5563e0d966c1 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/e890df3a7eda4aeaa6ea5563e0d966c1 2023-04-08 02:54:10,271 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/93b0b7a4dac34519bc0900b5a8880c40 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/93b0b7a4dac34519bc0900b5a8880c40 2023-04-08 02:54:10,272 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/4abfff4b5b434458afab50099f879917 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/4abfff4b5b434458afab50099f879917 2023-04-08 02:54:10,274 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/42b03fff9cf24ee6a6ba18ffbf0bcee9 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/42b03fff9cf24ee6a6ba18ffbf0bcee9 2023-04-08 02:54:10,275 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/0f6bacf9c35143d5b8dcdb050ee6adc6 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/0f6bacf9c35143d5b8dcdb050ee6adc6 2023-04-08 02:54:10,277 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/a13d734e06134627b9fa867d93e375e9 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/a13d734e06134627b9fa867d93e375e9 2023-04-08 02:54:10,278 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/4f7fa2fdf403442a92bc498fe52f61cc to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/4f7fa2fdf403442a92bc498fe52f61cc 2023-04-08 02:54:10,280 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/f673339f1deb49ce867e88c66b695cfc to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/f673339f1deb49ce867e88c66b695cfc 2023-04-08 02:54:10,281 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/68c4f8d7e63649cbb4e39399a89aa711 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/68c4f8d7e63649cbb4e39399a89aa711 2023-04-08 02:54:10,283 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/d29902ce88d14223b87388b5e46c0718 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/d29902ce88d14223b87388b5e46c0718 2023-04-08 02:54:10,284 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/184fbecf1b3a421eb4c22809f4e00863 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/184fbecf1b3a421eb4c22809f4e00863 2023-04-08 02:54:10,285 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/b0139a9cd85744e7819e5877d2b5922d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/b0139a9cd85744e7819e5877d2b5922d 2023-04-08 02:54:10,287 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/23266f53b1a446449809fb0860947dda to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/23266f53b1a446449809fb0860947dda 2023-04-08 02:54:10,288 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/2433dc1c9dff4bf3833cd8ad71b89481 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/2433dc1c9dff4bf3833cd8ad71b89481 2023-04-08 02:54:10,289 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/3516c3443ed14d3cb00a60a0da88cdab to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/3516c3443ed14d3cb00a60a0da88cdab 2023-04-08 02:54:10,290 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/8a2dbcc82ab54ae9a29f9a8fed7af429 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/8a2dbcc82ab54ae9a29f9a8fed7af429 2023-04-08 02:54:10,292 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/fe66cc14c0464bde824062e94ecd0925 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/fe66cc14c0464bde824062e94ecd0925 2023-04-08 02:54:10,293 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/529eaa2966fc40a69adf1a3a0b997333 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/529eaa2966fc40a69adf1a3a0b997333 2023-04-08 02:54:10,294 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/d9a06eb1d6f640f3836a93c704aecdbf to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/d9a06eb1d6f640f3836a93c704aecdbf 2023-04-08 02:54:10,325 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/90730644cd40498e8aad1e01bd6a3c37, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/e9a1795d40e74cf5bcc889cbaf98c034, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/17d35ecf0f8e4f73913d8069fb96909a, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/dd985b26bff540cc98b7d1610b82fc85, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/b8a752c44d30432ba24036bdb1220c3d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/b9da0693bc53472ea93f89e46e0c77cf, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/2e69ec9c942c41c9a7f01320bca92160, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/ef8fb2c75d714df19926c27c8e058aa5, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/6fb1dce0e3a14cc88e93d192f0baf7cb, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/b14d090365884d4a9c24e4d282935e9b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/bd358646e2a444fe92a07cc014e61202, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/93be6ee218604988bb2f770e1d61860f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/c9f2fa8a61b1418b9b59759c0901caf5, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/c3858b3e688e45e483b5b0de8835a937, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/1b34b86b131f40018e17518fc1c422db, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/ffea607b14e74d33838cd3974cb74f21, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/4d3995b9fcb0422a80bef86c7e9d93f1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/e34ff3199c964bc7920db8ed8315703a, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/83e6157e3f8a45a093313490e95044dc, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/088d6a9f1f8d47c49f62e84a7fbbb004, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/a28c63f412c54690bea95ba60f243357, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/4ac37fa2ad874e92b8cccb0e52f96d39, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/f7fcc74134604b59aec41abc6a122fc8, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/5f3da0869d02414db61448d84c2afb69, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/230365b62df54c64908c57482813f7c2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/2483fee87e024cc785f904ff5525d371, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/b33bafe0044c48dabe3acce857aae963, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/76969ff8ecfc495db08ba6f7400d3995, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/dabca8847a864d4393025973d67cf5bd, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/2df7062a8d4b4e9db7207c1bfe71997f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/732cfc5a06074905b33cd4db6110a782, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/86e2fdd4f57c48bfb7ae9f39ac81a068, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/749ff7dca1e8426eacbabcfc74c3aa8a, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/2be041c5e72b4397b2ff43e0f73bfbb7, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/6d74bc5b6cf9445da93626fb72573213, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/a9c1ce441957472b8ed15e63031a4398, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/1faf3da9d5c9401b810abf8fb7657d02, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/be1b0891b7444161afac6d47e68c875d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/422ec34edb5446619de7d2889e332c44, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/3c4f420fc9974bd08722ab2a7ef6b231, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/aa6e18bdc15b4625b0cc7435bd8e0f88, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/c09391f2298d41ff9085b3a33a158001] to archive 2023-04-08 02:54:10,326 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 02:54:10,328 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/90730644cd40498e8aad1e01bd6a3c37 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/90730644cd40498e8aad1e01bd6a3c37 2023-04-08 02:54:10,331 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/e9a1795d40e74cf5bcc889cbaf98c034 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/e9a1795d40e74cf5bcc889cbaf98c034 2023-04-08 02:54:10,332 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/17d35ecf0f8e4f73913d8069fb96909a to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/17d35ecf0f8e4f73913d8069fb96909a 2023-04-08 02:54:10,334 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/dd985b26bff540cc98b7d1610b82fc85 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/dd985b26bff540cc98b7d1610b82fc85 2023-04-08 02:54:10,336 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/b8a752c44d30432ba24036bdb1220c3d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/b8a752c44d30432ba24036bdb1220c3d 2023-04-08 02:54:10,339 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/b9da0693bc53472ea93f89e46e0c77cf to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/b9da0693bc53472ea93f89e46e0c77cf 2023-04-08 02:54:10,341 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/2e69ec9c942c41c9a7f01320bca92160 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/2e69ec9c942c41c9a7f01320bca92160 2023-04-08 02:54:10,343 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/ef8fb2c75d714df19926c27c8e058aa5 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/ef8fb2c75d714df19926c27c8e058aa5 2023-04-08 02:54:10,344 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/6fb1dce0e3a14cc88e93d192f0baf7cb to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/6fb1dce0e3a14cc88e93d192f0baf7cb 2023-04-08 02:54:10,345 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/b14d090365884d4a9c24e4d282935e9b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/b14d090365884d4a9c24e4d282935e9b 2023-04-08 02:54:10,347 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/bd358646e2a444fe92a07cc014e61202 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/bd358646e2a444fe92a07cc014e61202 2023-04-08 02:54:10,348 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/93be6ee218604988bb2f770e1d61860f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/93be6ee218604988bb2f770e1d61860f 2023-04-08 02:54:10,349 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/c9f2fa8a61b1418b9b59759c0901caf5 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/c9f2fa8a61b1418b9b59759c0901caf5 2023-04-08 02:54:10,350 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/c3858b3e688e45e483b5b0de8835a937 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/c3858b3e688e45e483b5b0de8835a937 2023-04-08 02:54:10,351 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/1b34b86b131f40018e17518fc1c422db to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/1b34b86b131f40018e17518fc1c422db 2023-04-08 02:54:10,352 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/ffea607b14e74d33838cd3974cb74f21 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/ffea607b14e74d33838cd3974cb74f21 2023-04-08 02:54:10,353 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/4d3995b9fcb0422a80bef86c7e9d93f1 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/4d3995b9fcb0422a80bef86c7e9d93f1 2023-04-08 02:54:10,355 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/e34ff3199c964bc7920db8ed8315703a to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/e34ff3199c964bc7920db8ed8315703a 2023-04-08 02:54:10,356 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/83e6157e3f8a45a093313490e95044dc to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/83e6157e3f8a45a093313490e95044dc 2023-04-08 02:54:10,360 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/088d6a9f1f8d47c49f62e84a7fbbb004 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/088d6a9f1f8d47c49f62e84a7fbbb004 2023-04-08 02:54:10,361 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/a28c63f412c54690bea95ba60f243357 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/a28c63f412c54690bea95ba60f243357 2023-04-08 02:54:10,363 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/4ac37fa2ad874e92b8cccb0e52f96d39 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/4ac37fa2ad874e92b8cccb0e52f96d39 2023-04-08 02:54:10,365 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/f7fcc74134604b59aec41abc6a122fc8 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/f7fcc74134604b59aec41abc6a122fc8 2023-04-08 02:54:10,368 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/5f3da0869d02414db61448d84c2afb69 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/5f3da0869d02414db61448d84c2afb69 2023-04-08 02:54:10,369 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/230365b62df54c64908c57482813f7c2 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/230365b62df54c64908c57482813f7c2 2023-04-08 02:54:10,371 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/2483fee87e024cc785f904ff5525d371 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/2483fee87e024cc785f904ff5525d371 2023-04-08 02:54:10,372 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/b33bafe0044c48dabe3acce857aae963 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/b33bafe0044c48dabe3acce857aae963 2023-04-08 02:54:10,374 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/76969ff8ecfc495db08ba6f7400d3995 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/76969ff8ecfc495db08ba6f7400d3995 2023-04-08 02:54:10,375 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/dabca8847a864d4393025973d67cf5bd to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/dabca8847a864d4393025973d67cf5bd 2023-04-08 02:54:10,376 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/2df7062a8d4b4e9db7207c1bfe71997f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/2df7062a8d4b4e9db7207c1bfe71997f 2023-04-08 02:54:10,377 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/732cfc5a06074905b33cd4db6110a782 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/732cfc5a06074905b33cd4db6110a782 2023-04-08 02:54:10,379 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/86e2fdd4f57c48bfb7ae9f39ac81a068 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/86e2fdd4f57c48bfb7ae9f39ac81a068 2023-04-08 02:54:10,380 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/749ff7dca1e8426eacbabcfc74c3aa8a to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/749ff7dca1e8426eacbabcfc74c3aa8a 2023-04-08 02:54:10,382 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/2be041c5e72b4397b2ff43e0f73bfbb7 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/2be041c5e72b4397b2ff43e0f73bfbb7 2023-04-08 02:54:10,384 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/6d74bc5b6cf9445da93626fb72573213 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/6d74bc5b6cf9445da93626fb72573213 2023-04-08 02:54:10,393 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/a9c1ce441957472b8ed15e63031a4398 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/a9c1ce441957472b8ed15e63031a4398 2023-04-08 02:54:10,395 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/1faf3da9d5c9401b810abf8fb7657d02 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/1faf3da9d5c9401b810abf8fb7657d02 2023-04-08 02:54:10,400 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/be1b0891b7444161afac6d47e68c875d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/be1b0891b7444161afac6d47e68c875d 2023-04-08 02:54:10,403 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/422ec34edb5446619de7d2889e332c44 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/422ec34edb5446619de7d2889e332c44 2023-04-08 02:54:10,405 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/3c4f420fc9974bd08722ab2a7ef6b231 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/3c4f420fc9974bd08722ab2a7ef6b231 2023-04-08 02:54:10,406 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/aa6e18bdc15b4625b0cc7435bd8e0f88 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/aa6e18bdc15b4625b0cc7435bd8e0f88 2023-04-08 02:54:10,408 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/c09391f2298d41ff9085b3a33a158001 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/c09391f2298d41ff9085b3a33a158001 2023-04-08 02:54:10,412 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/17d3302666834341a335679dc15f7e10, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/05aa5b6366f94252907e1836ac22f4f1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/69cd0d2bf2cf46cdac670c0e2a202d09, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/e996d940d4cd49babd94a6f624810b59, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/2d5ebf6211af4960a132e56f6f0c2193, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/fbef1f516d8844b49999a61d16459d8f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/6ec680ae83404cb781bd8802c32badee, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/2752988baa374592a6d98020fd24f2db, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/96aa0093de3242d5a26a998cef5faab7, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/f07537a5786c46bfb0b8a20caddb61e3, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/b9866a1a04764d72974b273dd733a90b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/065d01042c614202843c740fdb4c1b71, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/49b116944fcf4d17a7b0275b236bab03, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/7922ae0f305f4d99aeca9b72beb93be4, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/df79a0352066452a9ad7487c71d92604, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/652b796c688b45df9af805c1fe978dd4, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/55fc33fdaf6a4e3c9cc7a6fff56bfd5b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/a0cff0586fa14d48a0fdea5cf970ce3d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/81de0a9a3a36436bb52e1c178a5fb0d9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/914d60c335c84bdbaf05d3ae29c36994, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/db9959e359884f6db3e08dc9691c9989, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/0a323285bf2b49e384581a100f36e5f6, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/a5355e7e85a64193aac1abcb12dc1b0c, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/9e757487ae2542059992d6c55e562919, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/2c4fa4b2355740fc9456df420f6897f1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/a6147ebfa0594798b6e649b570fd6947, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/3a7b578c6f854610a634497e4b0e78b3, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/35d50f4578d34430a8bab026cd78fbd2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/f482297469d644dd8f790fe1480cd675, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/c71661957d354b6e892986d9cb75851c, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/4abc3ec196fb4dcdad7d251caa2c1996, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/5ef8c6ef672a4529a3070cb83360f9a9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/ab0cb190cf824b72996ffdf36eeff776, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/aa80eef02f854e19beb168b84fca968f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/9bacd3b806fb4d24b13833280353c163, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/b10fcaa5e9f84ef4bac1001fb91f76e2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/25e9af7fea7d4379831a609a7df98ee3, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/c515adae0a8e43ee9bf6da2ebc545814, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/1d0d3d80b29949399d8384528dda7936, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/5ecd08f0204345d1aacfd96a2417a924, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/0655708188504148b47f117b770e402d] to archive 2023-04-08 02:54:10,413 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 02:54:10,415 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/17d3302666834341a335679dc15f7e10 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/17d3302666834341a335679dc15f7e10 2023-04-08 02:54:10,416 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/05aa5b6366f94252907e1836ac22f4f1 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/05aa5b6366f94252907e1836ac22f4f1 2023-04-08 02:54:10,418 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/69cd0d2bf2cf46cdac670c0e2a202d09 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/69cd0d2bf2cf46cdac670c0e2a202d09 2023-04-08 02:54:10,419 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/e996d940d4cd49babd94a6f624810b59 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/e996d940d4cd49babd94a6f624810b59 2023-04-08 02:54:10,420 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/2d5ebf6211af4960a132e56f6f0c2193 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/2d5ebf6211af4960a132e56f6f0c2193 2023-04-08 02:54:10,422 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/fbef1f516d8844b49999a61d16459d8f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/fbef1f516d8844b49999a61d16459d8f 2023-04-08 02:54:10,423 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/6ec680ae83404cb781bd8802c32badee to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/6ec680ae83404cb781bd8802c32badee 2023-04-08 02:54:10,424 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/2752988baa374592a6d98020fd24f2db to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/2752988baa374592a6d98020fd24f2db 2023-04-08 02:54:10,426 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/96aa0093de3242d5a26a998cef5faab7 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/96aa0093de3242d5a26a998cef5faab7 2023-04-08 02:54:10,427 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/f07537a5786c46bfb0b8a20caddb61e3 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/f07537a5786c46bfb0b8a20caddb61e3 2023-04-08 02:54:10,429 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/b9866a1a04764d72974b273dd733a90b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/b9866a1a04764d72974b273dd733a90b 2023-04-08 02:54:10,430 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/065d01042c614202843c740fdb4c1b71 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/065d01042c614202843c740fdb4c1b71 2023-04-08 02:54:10,432 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/49b116944fcf4d17a7b0275b236bab03 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/49b116944fcf4d17a7b0275b236bab03 2023-04-08 02:54:10,433 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/7922ae0f305f4d99aeca9b72beb93be4 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/7922ae0f305f4d99aeca9b72beb93be4 2023-04-08 02:54:10,434 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/df79a0352066452a9ad7487c71d92604 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/df79a0352066452a9ad7487c71d92604 2023-04-08 02:54:10,436 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/652b796c688b45df9af805c1fe978dd4 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/652b796c688b45df9af805c1fe978dd4 2023-04-08 02:54:10,437 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/55fc33fdaf6a4e3c9cc7a6fff56bfd5b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/55fc33fdaf6a4e3c9cc7a6fff56bfd5b 2023-04-08 02:54:10,438 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/a0cff0586fa14d48a0fdea5cf970ce3d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/a0cff0586fa14d48a0fdea5cf970ce3d 2023-04-08 02:54:10,439 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/81de0a9a3a36436bb52e1c178a5fb0d9 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/81de0a9a3a36436bb52e1c178a5fb0d9 2023-04-08 02:54:10,441 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/914d60c335c84bdbaf05d3ae29c36994 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/914d60c335c84bdbaf05d3ae29c36994 2023-04-08 02:54:10,443 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/db9959e359884f6db3e08dc9691c9989 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/db9959e359884f6db3e08dc9691c9989 2023-04-08 02:54:10,444 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/0a323285bf2b49e384581a100f36e5f6 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/0a323285bf2b49e384581a100f36e5f6 2023-04-08 02:54:10,446 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/a5355e7e85a64193aac1abcb12dc1b0c to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/a5355e7e85a64193aac1abcb12dc1b0c 2023-04-08 02:54:10,447 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/9e757487ae2542059992d6c55e562919 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/9e757487ae2542059992d6c55e562919 2023-04-08 02:54:10,449 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/2c4fa4b2355740fc9456df420f6897f1 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/2c4fa4b2355740fc9456df420f6897f1 2023-04-08 02:54:10,450 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/a6147ebfa0594798b6e649b570fd6947 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/a6147ebfa0594798b6e649b570fd6947 2023-04-08 02:54:10,451 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/3a7b578c6f854610a634497e4b0e78b3 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/3a7b578c6f854610a634497e4b0e78b3 2023-04-08 02:54:10,452 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/35d50f4578d34430a8bab026cd78fbd2 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/35d50f4578d34430a8bab026cd78fbd2 2023-04-08 02:54:10,453 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/f482297469d644dd8f790fe1480cd675 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/f482297469d644dd8f790fe1480cd675 2023-04-08 02:54:10,455 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/c71661957d354b6e892986d9cb75851c to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/c71661957d354b6e892986d9cb75851c 2023-04-08 02:54:10,456 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/4abc3ec196fb4dcdad7d251caa2c1996 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/4abc3ec196fb4dcdad7d251caa2c1996 2023-04-08 02:54:10,457 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/5ef8c6ef672a4529a3070cb83360f9a9 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/5ef8c6ef672a4529a3070cb83360f9a9 2023-04-08 02:54:10,459 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/ab0cb190cf824b72996ffdf36eeff776 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/ab0cb190cf824b72996ffdf36eeff776 2023-04-08 02:54:10,460 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/aa80eef02f854e19beb168b84fca968f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/aa80eef02f854e19beb168b84fca968f 2023-04-08 02:54:10,462 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/9bacd3b806fb4d24b13833280353c163 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/9bacd3b806fb4d24b13833280353c163 2023-04-08 02:54:10,464 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/b10fcaa5e9f84ef4bac1001fb91f76e2 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/b10fcaa5e9f84ef4bac1001fb91f76e2 2023-04-08 02:54:10,465 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/25e9af7fea7d4379831a609a7df98ee3 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/25e9af7fea7d4379831a609a7df98ee3 2023-04-08 02:54:10,466 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/c515adae0a8e43ee9bf6da2ebc545814 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/c515adae0a8e43ee9bf6da2ebc545814 2023-04-08 02:54:10,467 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/1d0d3d80b29949399d8384528dda7936 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/1d0d3d80b29949399d8384528dda7936 2023-04-08 02:54:10,468 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/5ecd08f0204345d1aacfd96a2417a924 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/5ecd08f0204345d1aacfd96a2417a924 2023-04-08 02:54:10,472 DEBUG [StoreCloser-TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/0655708188504148b47f117b770e402d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/0655708188504148b47f117b770e402d 2023-04-08 02:54:10,478 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/recovered.edits/883.seqid, newMaxSeqId=883, maxSeqId=1 2023-04-08 02:54:10,480 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415. 2023-04-08 02:54:10,480 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 9a2d8d09b1a5f9cc567f207bd50b4415: 2023-04-08 02:54:10,483 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:54:10,483 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=13 updating hbase:meta row=9a2d8d09b1a5f9cc567f207bd50b4415, regionState=CLOSED 2023-04-08 02:54:10,483 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922450483"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680922450483"}]},"ts":"1680922450483"} 2023-04-08 02:54:10,487 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=14, resume processing ppid=13 2023-04-08 02:54:10,488 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=14, ppid=13, state=SUCCESS; CloseRegionProcedure 9a2d8d09b1a5f9cc567f207bd50b4415, server=jenkins-hbase4.apache.org,35089,1680922417983 in 1.7010 sec 2023-04-08 02:54:10,490 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=13, resume processing ppid=12 2023-04-08 02:54:10,491 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=13, ppid=12, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=9a2d8d09b1a5f9cc567f207bd50b4415, UNASSIGN in 1.7080 sec 2023-04-08 02:54:10,491 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680922450491"}]},"ts":"1680922450491"} 2023-04-08 02:54:10,493 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-08 02:54:10,495 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-08 02:54:10,498 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=12, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.7290 sec 2023-04-08 02:54:10,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=12 2023-04-08 02:54:10,881 INFO [Listener at localhost/39741] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 12 completed 2023-04-08 02:54:10,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-08 02:54:10,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure2.ProcedureExecutor(1029): Stored pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:54:10,894 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=15, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:54:10,895 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=15, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:54:10,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-08 02:54:10,900 DEBUG [HFileArchiver-3] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:54:10,903 DEBUG [HFileArchiver-3] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A, FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B, FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C, FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/recovered.edits] 2023-04-08 02:54:10,907 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/188ef647a35142ce96f16f29d6ec20d7 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/188ef647a35142ce96f16f29d6ec20d7 2023-04-08 02:54:10,908 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/434c49ee61054dc78871454993f38627 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/434c49ee61054dc78871454993f38627 2023-04-08 02:54:10,910 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/8d97c1b954094d0f8ebbce47a1cfd753 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/A/8d97c1b954094d0f8ebbce47a1cfd753 2023-04-08 02:54:10,913 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/0c653dd45e16493abd81bd0eb92ee579 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/0c653dd45e16493abd81bd0eb92ee579 2023-04-08 02:54:10,914 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/6eb1c0f7995640d6b5bff486da173f29 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/6eb1c0f7995640d6b5bff486da173f29 2023-04-08 02:54:10,916 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/c6e4135280fc4c71bf5e910289e5517f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/B/c6e4135280fc4c71bf5e910289e5517f 2023-04-08 02:54:10,919 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/236cdc0742f64fcd82006df3e370b148 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/236cdc0742f64fcd82006df3e370b148 2023-04-08 02:54:10,920 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/529dc5f2b6464be4ae55bcfb004fbd32 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/529dc5f2b6464be4ae55bcfb004fbd32 2023-04-08 02:54:10,922 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/e7b505b094dc400bae3fb45391bf8d66 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/C/e7b505b094dc400bae3fb45391bf8d66 2023-04-08 02:54:10,926 DEBUG [HFileArchiver-3] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/recovered.edits/883.seqid to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415/recovered.edits/883.seqid 2023-04-08 02:54:10,927 DEBUG [HFileArchiver-3] backup.HFileArchiver(596): Deleted hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/9a2d8d09b1a5f9cc567f207bd50b4415 2023-04-08 02:54:10,927 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-08 02:54:10,929 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=15, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:54:10,950 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-08 02:54:10,993 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-08 02:54:10,994 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=15, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:54:10,994 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-08 02:54:10,995 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680922450994"}]},"ts":"9223372036854775807"} 2023-04-08 02:54:10,997 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-08 02:54:10,997 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 9a2d8d09b1a5f9cc567f207bd50b4415, NAME => 'TestAcidGuarantees,,1680922422620.9a2d8d09b1a5f9cc567f207bd50b4415.', STARTKEY => '', ENDKEY => ''}] 2023-04-08 02:54:10,997 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-08 02:54:10,997 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680922450997"}]},"ts":"9223372036854775807"} 2023-04-08 02:54:10,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-08 02:54:10,999 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-08 02:54:11,001 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=15, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:54:11,002 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=15, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 114 msec 2023-04-08 02:54:11,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=15 2023-04-08 02:54:11,199 INFO [Listener at localhost/39741] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 15 completed 2023-04-08 02:54:11,215 INFO [Listener at localhost/39741] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMixedAtomicity Thread=233 (was 207) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-12 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS-EventLoopGroup-1-3 org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native Method) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:209) org.apache.hbase.thirdparty.io.netty.channel.epoll.Native.epollWait(Native.java:202) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.epollWaitNoTimerChange(EpollEventLoop.java:306) org.apache.hbase.thirdparty.io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:363) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-4 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-11 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: Timer for 'HBase' metrics system java.lang.Object.wait(Native Method) java.util.TimerThread.mainLoop(Timer.java:552) java.util.TimerThread.run(Timer.java:505) Potentially hanging thread: RS:0;jenkins-hbase4:35089-shortCompactions-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.PriorityBlockingQueue.take(PriorityBlockingQueue.java:549) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: (jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.SynchronousQueue$TransferStack.awaitFulfill(SynchronousQueue.java:458) java.util.concurrent.SynchronousQueue$TransferStack.transfer(SynchronousQueue.java:362) java.util.concurrent.SynchronousQueue.take(SynchronousQueue.java:924) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d1014ec-shared-pool-14 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-9 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-10 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-6 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d1014ec-shared-pool-15 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-5 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: DataXceiver for client DFSClient_NONMAPREDUCE_-749670039_15 at /127.0.0.1:52084 [Waiting for operation #3] sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hadoop.net.SocketIOWithTimeout$SelectorPool.select(SocketIOWithTimeout.java:335) org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:157) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161) org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131) java.io.BufferedInputStream.fill(BufferedInputStream.java:246) java.io.BufferedInputStream.read(BufferedInputStream.java:265) java.io.DataInputStream.readShort(DataInputStream.java:312) org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.readOp(Receiver.java:67) org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:269) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d1014ec-shared-pool-16 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: hconnection-0x2d1014ec-shared-pool-17 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-7 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-13 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: HFileArchiver-3 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-3 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-2 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:62) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:883) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RSProcedureDispatcher-pool-2 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078) java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:467) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1073) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RPCClient-NioEventLoopGroup-4-8 sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) org.apache.hbase.thirdparty.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:879) org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:526) org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) java.lang.Thread.run(Thread.java:750) Potentially hanging thread: RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5 sun.misc.Unsafe.park(Native Method) java.util.concurrent.locks.LockSupport.park(LockSupport.java:175) java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039) java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442) java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074) java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134) java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) java.lang.Thread.run(Thread.java:750) - Thread LEAK? -, OpenFileDescriptor=569 (was 470) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=424 (was 336) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 171), AvailableMemoryMB=1790 (was 2356) 2023-04-08 02:54:11,226 INFO [Listener at localhost/39741] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=233, OpenFileDescriptor=569, MaxFileDescriptor=60000, SystemLoadAverage=424, ProcessCount=170, AvailableMemoryMB=1789 2023-04-08 02:54:11,228 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-08 02:54:11,229 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-08 02:54:11,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure2.ProcedureExecutor(1029): Stored pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-08 02:54:11,231 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-08 02:54:11,231 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 16 2023-04-08 02:54:11,232 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-08 02:54:11,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-08 02:54:11,233 DEBUG [HFileArchiver-4] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:11,234 DEBUG [HFileArchiver-4] backup.HFileArchiver(153): Directory hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d empty. 2023-04-08 02:54:11,234 DEBUG [HFileArchiver-4] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:11,235 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-08 02:54:11,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-08 02:54:11,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-08 02:54:11,647 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-08 02:54:11,649 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => d38b4e710235c58cc9479560817e4c8d, NAME => 'TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp 2023-04-08 02:54:11,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-08 02:54:12,057 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 02:54:12,058 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing d38b4e710235c58cc9479560817e4c8d, disabling compactions & flushes 2023-04-08 02:54:12,058 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:12,058 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:12,058 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. after waiting 0 ms 2023-04-08 02:54:12,058 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:12,058 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:12,058 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:12,060 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-08 02:54:12,061 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922452061"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680922452061"}]},"ts":"1680922452061"} 2023-04-08 02:54:12,062 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-08 02:54:12,063 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-08 02:54:12,063 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680922452063"}]},"ts":"1680922452063"} 2023-04-08 02:54:12,064 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-08 02:54:12,067 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d38b4e710235c58cc9479560817e4c8d, ASSIGN}] 2023-04-08 02:54:12,068 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d38b4e710235c58cc9479560817e4c8d, ASSIGN 2023-04-08 02:54:12,069 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=17, ppid=16, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=d38b4e710235c58cc9479560817e4c8d, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35089,1680922417983; forceNewPlan=false, retain=false 2023-04-08 02:54:12,220 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=d38b4e710235c58cc9479560817e4c8d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:12,221 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922452220"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680922452220"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680922452220"}]},"ts":"1680922452220"} 2023-04-08 02:54:12,222 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=18, ppid=17, state=RUNNABLE; OpenRegionProcedure d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983}] 2023-04-08 02:54:12,336 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-08 02:54:12,378 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:12,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d38b4e710235c58cc9479560817e4c8d, NAME => 'TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.', STARTKEY => '', ENDKEY => ''} 2023-04-08 02:54:12,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:12,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 02:54:12,378 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:12,379 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:12,380 INFO [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:12,381 DEBUG [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A 2023-04-08 02:54:12,381 DEBUG [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A 2023-04-08 02:54:12,381 INFO [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 02:54:12,382 INFO [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d38b4e710235c58cc9479560817e4c8d columnFamilyName A 2023-04-08 02:54:12,382 INFO [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] regionserver.HStore(310): Store=d38b4e710235c58cc9479560817e4c8d/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:54:12,382 INFO [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:12,383 DEBUG [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B 2023-04-08 02:54:12,383 DEBUG [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B 2023-04-08 02:54:12,384 INFO [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 02:54:12,384 INFO [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d38b4e710235c58cc9479560817e4c8d columnFamilyName B 2023-04-08 02:54:12,384 INFO [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] regionserver.HStore(310): Store=d38b4e710235c58cc9479560817e4c8d/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:54:12,384 INFO [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:12,385 DEBUG [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C 2023-04-08 02:54:12,385 DEBUG [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C 2023-04-08 02:54:12,385 INFO [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 02:54:12,386 INFO [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d38b4e710235c58cc9479560817e4c8d columnFamilyName C 2023-04-08 02:54:12,386 INFO [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] regionserver.HStore(310): Store=d38b4e710235c58cc9479560817e4c8d/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:54:12,386 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:12,387 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:12,387 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:12,389 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-08 02:54:12,390 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:12,391 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-08 02:54:12,392 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened d38b4e710235c58cc9479560817e4c8d; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=68703089, jitterRate=0.02375580370426178}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-08 02:54:12,392 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:12,392 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d., pid=18, masterSystemTime=1680922452374 2023-04-08 02:54:12,394 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:12,394 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:12,394 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=17 updating hbase:meta row=d38b4e710235c58cc9479560817e4c8d, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:12,395 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922452394"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680922452394"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680922452394"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680922452394"}]},"ts":"1680922452394"} 2023-04-08 02:54:12,397 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=18, resume processing ppid=17 2023-04-08 02:54:12,397 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=18, ppid=17, state=SUCCESS; OpenRegionProcedure d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 in 174 msec 2023-04-08 02:54:12,398 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=17, resume processing ppid=16 2023-04-08 02:54:12,398 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=17, ppid=16, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d38b4e710235c58cc9479560817e4c8d, ASSIGN in 330 msec 2023-04-08 02:54:12,399 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-08 02:54:12,399 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680922452399"}]},"ts":"1680922452399"} 2023-04-08 02:54:12,400 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-08 02:54:12,402 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=16, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-08 02:54:12,403 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=16, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1730 sec 2023-04-08 02:54:13,337 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=16 2023-04-08 02:54:13,338 INFO [Listener at localhost/39741] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 16 completed 2023-04-08 02:54:13,339 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x52274650 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:54:13,343 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7829b43e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:54:13,345 DEBUG [Listener at localhost/39741] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:54:13,349 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:35034, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:54:13,350 DEBUG [Listener at localhost/39741] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-08 02:54:13,353 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33798, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-08 02:54:13,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-08 02:54:13,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-08 02:54:13,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure2.ProcedureExecutor(1029): Stored pid=19, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-08 02:54:13,785 DEBUG [PEWorker-3] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-08 02:54:13,786 DEBUG [PEWorker-3] util.FSTableDescriptors(519): Deleted hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-08 02:54:13,786 INFO [PEWorker-3] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-08 02:54:13,790 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=20, ppid=19, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-08 02:54:13,803 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d38b4e710235c58cc9479560817e4c8d, REOPEN/MOVE}] 2023-04-08 02:54:13,803 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d38b4e710235c58cc9479560817e4c8d, REOPEN/MOVE 2023-04-08 02:54:13,804 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=d38b4e710235c58cc9479560817e4c8d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:13,804 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922453804"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680922453804"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680922453804"}]},"ts":"1680922453804"} 2023-04-08 02:54:13,806 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=22, ppid=21, state=RUNNABLE; CloseRegionProcedure d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983}] 2023-04-08 02:54:13,958 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:13,959 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing d38b4e710235c58cc9479560817e4c8d, disabling compactions & flushes 2023-04-08 02:54:13,959 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:13,959 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:13,959 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. after waiting 0 ms 2023-04-08 02:54:13,959 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:13,962 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-08 02:54:13,962 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:13,962 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:13,963 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: d38b4e710235c58cc9479560817e4c8d to self. 2023-04-08 02:54:13,964 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:13,964 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=d38b4e710235c58cc9479560817e4c8d, regionState=CLOSED 2023-04-08 02:54:13,965 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922453964"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680922453964"}]},"ts":"1680922453964"} 2023-04-08 02:54:13,967 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=22, resume processing ppid=21 2023-04-08 02:54:13,967 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=22, ppid=21, state=SUCCESS; CloseRegionProcedure d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 in 161 msec 2023-04-08 02:54:13,968 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=21, ppid=20, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=d38b4e710235c58cc9479560817e4c8d, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,35089,1680922417983; forceNewPlan=false, retain=true 2023-04-08 02:54:14,118 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=d38b4e710235c58cc9479560817e4c8d, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,118 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922454118"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680922454118"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680922454118"}]},"ts":"1680922454118"} 2023-04-08 02:54:14,120 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=23, ppid=21, state=RUNNABLE; OpenRegionProcedure d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983}] 2023-04-08 02:54:14,275 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:14,275 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d38b4e710235c58cc9479560817e4c8d, NAME => 'TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.', STARTKEY => '', ENDKEY => ''} 2023-04-08 02:54:14,275 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:14,275 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 02:54:14,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:14,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:14,278 INFO [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:14,279 DEBUG [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A 2023-04-08 02:54:14,279 DEBUG [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A 2023-04-08 02:54:14,279 INFO [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 02:54:14,282 INFO [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d38b4e710235c58cc9479560817e4c8d columnFamilyName A 2023-04-08 02:54:14,284 INFO [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] regionserver.HStore(310): Store=d38b4e710235c58cc9479560817e4c8d/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:54:14,284 INFO [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:14,285 DEBUG [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B 2023-04-08 02:54:14,285 DEBUG [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B 2023-04-08 02:54:14,285 INFO [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 02:54:14,286 INFO [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d38b4e710235c58cc9479560817e4c8d columnFamilyName B 2023-04-08 02:54:14,286 INFO [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] regionserver.HStore(310): Store=d38b4e710235c58cc9479560817e4c8d/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:54:14,286 INFO [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:14,287 DEBUG [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C 2023-04-08 02:54:14,287 DEBUG [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C 2023-04-08 02:54:14,287 INFO [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 02:54:14,288 INFO [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d38b4e710235c58cc9479560817e4c8d columnFamilyName C 2023-04-08 02:54:14,288 INFO [StoreOpener-d38b4e710235c58cc9479560817e4c8d-1] regionserver.HStore(310): Store=d38b4e710235c58cc9479560817e4c8d/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:54:14,288 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:14,289 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:14,290 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:14,291 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-08 02:54:14,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:14,293 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened d38b4e710235c58cc9479560817e4c8d; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=60066569, jitterRate=-0.10493837296962738}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-08 02:54:14,293 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:14,294 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d., pid=23, masterSystemTime=1680922454272 2023-04-08 02:54:14,295 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:14,296 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:14,296 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=21 updating hbase:meta row=d38b4e710235c58cc9479560817e4c8d, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,296 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922454296"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680922454296"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680922454296"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680922454296"}]},"ts":"1680922454296"} 2023-04-08 02:54:14,299 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=23, resume processing ppid=21 2023-04-08 02:54:14,299 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=23, ppid=21, state=SUCCESS; OpenRegionProcedure d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 in 177 msec 2023-04-08 02:54:14,300 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=21, resume processing ppid=20 2023-04-08 02:54:14,300 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=21, ppid=20, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d38b4e710235c58cc9479560817e4c8d, REOPEN/MOVE in 496 msec 2023-04-08 02:54:14,303 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=20, resume processing ppid=19 2023-04-08 02:54:14,303 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=20, ppid=19, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 512 msec 2023-04-08 02:54:14,304 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=19, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 939 msec 2023-04-08 02:54:14,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=19 2023-04-08 02:54:14,313 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x34135972 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:54:14,318 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@33a47c05, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:54:14,319 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x02bab6b2 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:54:14,322 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1441c9f0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:54:14,323 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x24bd3872 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:54:14,326 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@73e51571, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:54:14,327 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x370947cb to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:54:14,333 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6c1c92c6, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:54:14,333 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x210d1008 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:54:14,336 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c472cfd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:54:14,336 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x498694cb to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:54:14,340 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c4379c2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:54:14,340 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x421e5356 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:54:14,343 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3f43b712, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:54:14,344 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x4a0fd7b4 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:54:14,349 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7a3bb940, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:54:14,349 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x1a7b66a3 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:54:14,353 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2efdb02f, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:54:14,359 DEBUG [hconnection-0x32f0b4b0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:54:14,359 DEBUG [hconnection-0x141ee72d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:54:14,361 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57992, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:54:14,361 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:57996, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:54:14,363 DEBUG [hconnection-0x12051855-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:54:14,366 DEBUG [hconnection-0x69d4ad36-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:54:14,366 DEBUG [hconnection-0x5cdb4c18-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:54:14,367 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58012, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:54:14,371 DEBUG [hconnection-0x4fe6e4ad-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:54:14,372 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58006, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:54:14,372 DEBUG [hconnection-0x3a231f0-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:54:14,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:14,380 DEBUG [hconnection-0x4390ee54-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:54:14,380 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58024, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:54:14,380 DEBUG [hconnection-0x5d7da072-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:54:14,382 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58050, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:54:14,381 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58036, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:54:14,385 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58068, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:54:14,385 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:58052, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:54:14,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:14,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:14,411 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:14,411 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:14,411 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:14,411 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:14,408 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d38b4e710235c58cc9479560817e4c8d 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-08 02:54:14,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:14,412 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,412 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:14,413 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:14,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:14,413 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:14,413 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,413 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:14,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:14,414 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:14,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:14,414 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:14,415 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:14,415 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:14,415 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:14,415 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:14,415 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:14,415 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=A 2023-04-08 02:54:14,415 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:14,415 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:14,415 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:14,415 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:14,416 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:14,416 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:14,416 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:14,416 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:14,416 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,416 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:14,416 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:14,415 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:14,416 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=B 2023-04-08 02:54:14,416 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:14,416 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=C 2023-04-08 02:54:14,417 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:14,418 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,418 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:14,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:14,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:14,419 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:14,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:14,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,419 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:14,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:14,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:14,423 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16e7805b[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:14,423 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:14,423 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,426 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,426 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,426 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:14,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,428 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:14,428 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:14,428 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:14,429 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:14,429 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:14,429 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:14,430 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:14,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:14,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:14,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:14,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:14,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:14,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:14,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:14,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:14,434 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:14,434 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:14,434 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:14,435 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:14,435 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:14,436 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:14,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:14,439 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,440 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:14,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:14,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:14,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:14,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,443 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:14,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:14,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:14,444 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:14,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:14,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922514450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:14,454 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:14,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57996 deadline: 1680922514451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,455 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:14,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 6 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922514452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,456 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,456 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:14,456 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:14,456 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:14,456 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:14,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:14,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:14,457 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:14,457 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,457 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:14,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:14,457 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,457 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,457 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:14,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:14,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:14,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:14,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922514456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:14,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922514459, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,462 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:14,462 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:14,480 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408ee72836b52a547f197a3631e87d0405d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408ee72836b52a547f197a3631e87d0405d 2023-04-08 02:54:14,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:14,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922514557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,558 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:14,558 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:14,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57996 deadline: 1680922514557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922514557, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,560 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:14,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922514560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:14,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922514561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,563 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:14,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:14,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:14,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:14,589 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:14,589 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:14,589 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:14,589 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:14,589 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:14,590 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,590 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:14,591 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,592 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,592 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:14,597 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:14,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:14,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:14,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:14,606 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:14,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:14,635 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:14,636 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:14,636 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:14,636 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:14,636 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:14,636 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:14,636 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:14,638 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:14,638 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:14,638 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:14,639 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:14,639 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:14,639 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:14,639 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,639 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:14,639 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:14,643 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:14,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:14,644 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:14,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:14,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:14,644 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,645 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:14,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:14,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:14,646 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1431bd87[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:14,646 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:14,646 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,648 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,648 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,649 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:14,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:14,649 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:14,649 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:14,650 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:14,650 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:14,650 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:14,650 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:14,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:14,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:14,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:14,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:14,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:14,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:14,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:14,654 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:14,654 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:14,654 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:14,654 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:14,654 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:14,656 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,656 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:14,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,656 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:14,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:14,656 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:14,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:14,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:14,659 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:14,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:14,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,664 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:14,664 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:14,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:14,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:14,665 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,665 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:14,665 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:14,665 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,665 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,665 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:14,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:14,665 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:14,665 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,665 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:14,665 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:14,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:14,667 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:14,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:14,762 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:14,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:14,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922514761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922514761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:14,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922514761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,765 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:14,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922514764, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,766 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:14,767 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:14,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57996 deadline: 1680922514765, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:14,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:14,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:14,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:14,796 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:14,796 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:14,796 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:14,796 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:14,797 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,797 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:14,799 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:14,799 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:14,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:14,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:14,800 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,800 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,800 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:14,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:14,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:14,800 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:14,801 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:14,801 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:14,801 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:14,801 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:14,801 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:14,801 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:14,801 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:14,801 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:14,801 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:14,802 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:14,802 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:14,802 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:14,802 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:14,802 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,802 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:14,802 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:14,804 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:14,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:14,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:14,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:14,805 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:14,806 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,806 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:14,806 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:14,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:14,807 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@970c979[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:14,807 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:14,807 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,809 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:14,809 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:14,809 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:14,809 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,809 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:14,811 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:14,811 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:14,811 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:14,811 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:14,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:14,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:14,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:14,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:14,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:14,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:14,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:14,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:14,815 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:14,815 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:14,815 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:14,815 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:14,815 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:14,817 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,817 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:14,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,817 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:14,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:14,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:14,817 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:14,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:14,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:14,819 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:14,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:14,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,822 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,822 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:14,822 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:14,822 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:14,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:14,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:14,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:14,822 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,822 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:14,823 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:14,822 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,823 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,823 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,823 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:14,823 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:14,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:14,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:14,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:14,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:14,899 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=18, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/080a10550c5a417e880d276fa08a8dbe 2023-04-08 02:54:14,923 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:14,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:14,927 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:14,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:14,944 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:14,944 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:14,944 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:14,944 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:14,944 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:14,945 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,945 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:14,946 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:14,946 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:14,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:14,946 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:14,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,947 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,947 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:14,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:14,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:14,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:14,960 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:14,960 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:14,960 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:14,960 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:14,960 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:14,961 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:14,961 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:14,961 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:14,962 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:14,962 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:14,962 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:14,962 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:14,962 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:14,962 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,962 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:14,962 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:14,968 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,968 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:14,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:14,969 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,969 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:14,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:14,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:14,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:14,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:14,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:14,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@755b7bb[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:14,971 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:14,971 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,972 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,972 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:14,972 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:14,972 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:14,972 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,972 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:14,975 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:14,975 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:14,975 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:14,975 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:14,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:14,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:14,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:14,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:14,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:14,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:14,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:14,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:14,978 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:14,978 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:14,978 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:14,978 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:14,978 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:14,980 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:14,980 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:14,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,980 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:14,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:14,981 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:14,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:14,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:14,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:14,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:14,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,984 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:14,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:14,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,990 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:14,990 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:14,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:14,990 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:14,990 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:14,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:14,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:14,990 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:14,990 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:14,990 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:14,990 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,990 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:14,990 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:14,991 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:14,991 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:14,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:14,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:15,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:15,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922515065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,066 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:15,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922515065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,067 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:15,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922515066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,068 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:15,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922515067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:15,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57996 deadline: 1680922515073, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,091 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:15,091 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:15,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:15,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:15,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:15,095 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:15,095 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:15,095 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:15,095 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:15,096 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,096 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:15,097 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,097 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:15,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:15,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:15,098 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,098 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:15,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:15,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:15,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:15,099 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:15,100 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:15,100 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:15,100 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:15,100 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:15,100 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:15,100 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:15,100 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:15,100 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:15,100 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:15,100 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:15,100 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:15,100 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:15,100 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:15,100 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,100 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:15,100 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:15,102 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,102 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:15,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:15,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:15,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:15,103 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:15,103 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:15,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:15,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:15,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:15,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:15,104 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d9a0a52[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:15,104 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:15,104 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:15,105 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:15,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:15,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:15,105 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:15,105 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:15,105 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:15,106 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,106 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:15,108 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:15,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:15,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:15,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:15,109 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:15,109 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:15,109 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:15,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:15,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:15,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:15,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:15,112 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:15,112 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:15,112 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:15,112 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:15,112 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:15,112 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:15,114 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,114 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:15,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:15,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,115 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:15,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:15,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:15,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:15,115 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:15,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:15,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:15,117 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:15,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:15,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,121 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,121 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:15,121 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:15,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:15,121 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:15,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:15,121 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:15,121 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:15,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:15,121 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:15,121 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:15,121 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:15,121 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:15,121 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,122 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:15,122 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:15,122 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:15,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:15,221 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:15,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:15,225 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:15,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:15,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:15,226 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:15,226 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:15,226 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:15,226 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:15,227 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,227 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:15,229 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,232 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:15,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:15,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:15,240 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,240 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:15,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:15,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:15,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:15,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:15,244 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:15,244 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:15,244 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:15,244 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:15,244 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:15,256 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:15,256 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:15,264 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:15,264 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:15,264 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:15,264 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:15,264 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:15,264 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:15,264 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,264 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:15,264 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:15,266 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,266 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:15,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:15,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:15,266 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:15,266 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:15,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:15,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:15,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:15,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:15,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:15,268 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@34de57e3[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:15,268 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:15,268 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:15,269 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:15,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:15,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:15,270 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:15,270 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:15,270 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,270 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:15,270 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:15,271 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:15,271 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:15,271 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:15,271 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:15,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:15,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:15,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:15,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:15,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:15,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:15,276 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:15,276 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:15,276 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:15,276 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:15,276 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:15,276 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:15,278 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,278 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:15,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:15,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,279 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,280 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:15,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:15,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:15,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:15,282 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:15,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:15,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:15,284 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:15,284 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:15,284 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:15,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:15,284 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:15,284 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:15,285 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:15,285 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,285 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:15,285 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:15,285 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:15,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:15,285 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:15,285 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:15,286 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:15,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:15,286 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:15,364 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/72811cc7722144bab7f684d1cad2be1b 2023-04-08 02:54:15,385 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:15,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:15,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:15,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:15,393 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:15,393 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:15,393 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:15,393 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:15,393 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:15,394 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,394 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:15,396 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,396 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:15,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:15,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:15,396 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,396 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:15,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:15,398 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:15,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:15,398 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:15,400 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:15,400 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:15,400 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:15,400 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:15,400 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:15,403 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:15,403 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:15,403 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:15,403 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:15,403 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:15,403 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:15,403 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:15,404 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:15,404 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,404 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:15,404 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:15,406 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:15,406 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:15,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:15,406 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:15,406 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:15,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:15,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:15,410 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:15,410 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=18 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/0ff8bedb3705457fbe57b88601b98914 2023-04-08 02:54:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:15,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:15,411 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3b6a7b00[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:15,411 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:15,411 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:15,413 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:15,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:15,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:15,413 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:15,413 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:15,413 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:15,413 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,413 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:15,414 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:15,414 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,414 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:15,414 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:15,415 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:15,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:15,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,416 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/080a10550c5a417e880d276fa08a8dbe as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/080a10550c5a417e880d276fa08a8dbe 2023-04-08 02:54:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:15,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:15,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:15,419 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:15,419 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:15,419 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:15,419 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:15,419 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:15,419 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:15,422 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,422 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:15,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:15,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:15,422 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/080a10550c5a417e880d276fa08a8dbe, entries=250, sequenceid=18, filesize=38.7 K 2023-04-08 02:54:15,423 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:15,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:15,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:15,424 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/72811cc7722144bab7f684d1cad2be1b as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/72811cc7722144bab7f684d1cad2be1b 2023-04-08 02:54:15,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:15,424 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:15,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:15,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,425 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:15,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:15,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,432 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,432 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:15,432 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:15,432 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,432 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:15,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:15,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:15,432 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:15,432 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:15,433 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:15,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:15,433 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:15,432 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:15,433 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,433 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:15,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:15,433 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:15,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:15,434 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/72811cc7722144bab7f684d1cad2be1b, entries=150, sequenceid=18, filesize=11.6 K 2023-04-08 02:54:15,435 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/0ff8bedb3705457fbe57b88601b98914 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/0ff8bedb3705457fbe57b88601b98914 2023-04-08 02:54:15,451 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/0ff8bedb3705457fbe57b88601b98914, entries=150, sequenceid=18, filesize=11.6 K 2023-04-08 02:54:15,452 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for d38b4e710235c58cc9479560817e4c8d in 1044ms, sequenceid=18, compaction requested=false 2023-04-08 02:54:15,452 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:15,533 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:15,534 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:15,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:15,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:15,539 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:15,539 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:15,539 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:15,539 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:15,539 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:15,539 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,539 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:15,541 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:15,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:15,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:15,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:15,542 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:15,542 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:15,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:15,542 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:15,543 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:15,543 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:15,543 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,543 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:15,543 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:15,543 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:15,543 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:15,544 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:15,544 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:15,544 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:15,544 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:15,544 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d38b4e710235c58cc9479560817e4c8d 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-08 02:54:15,544 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=A 2023-04-08 02:54:15,545 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:15,545 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=B 2023-04-08 02:54:15,545 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:15,545 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=C 2023-04-08 02:54:15,545 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:15,570 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:15,570 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:15,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:15,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57996 deadline: 1680922515578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,583 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:15,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922515582, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:15,584 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:15,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922515583, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922515584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,584 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:15,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922515584, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,686 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:15,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922515685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:15,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922515685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:15,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922515685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:15,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922515685, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:15,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922515888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,889 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:15,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922515889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,891 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:15,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922515889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:15,897 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:15,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922515895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:16,004 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040812546a5c68b440d3ba9c785fd25a69e7 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040812546a5c68b440d3ba9c785fd25a69e7 2023-04-08 02:54:16,042 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-08 02:54:16,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:16,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922516191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:16,194 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:16,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922516193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:16,196 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:16,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922516196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:16,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:16,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922516201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:16,412 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=37, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/c6f6445e4bf14b2f9b7e03c4c4bbf616 2023-04-08 02:54:16,586 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:16,586 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57996 deadline: 1680922516586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:16,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:16,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922516696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:16,697 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:16,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922516696, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:16,701 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:16,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922516700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:16,709 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:16,709 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922516708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:16,829 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/97d505898f1f442c972b27c3a7badb61 2023-04-08 02:54:17,250 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=37 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/0bc82dc108204b4ba933b984799b2a92 2023-04-08 02:54:17,257 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/c6f6445e4bf14b2f9b7e03c4c4bbf616 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/c6f6445e4bf14b2f9b7e03c4c4bbf616 2023-04-08 02:54:17,262 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/c6f6445e4bf14b2f9b7e03c4c4bbf616, entries=150, sequenceid=37, filesize=25.1 K 2023-04-08 02:54:17,263 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/97d505898f1f442c972b27c3a7badb61 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/97d505898f1f442c972b27c3a7badb61 2023-04-08 02:54:17,267 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/97d505898f1f442c972b27c3a7badb61, entries=150, sequenceid=37, filesize=11.6 K 2023-04-08 02:54:17,268 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/0bc82dc108204b4ba933b984799b2a92 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/0bc82dc108204b4ba933b984799b2a92 2023-04-08 02:54:17,282 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/0bc82dc108204b4ba933b984799b2a92, entries=150, sequenceid=37, filesize=11.6 K 2023-04-08 02:54:17,283 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for d38b4e710235c58cc9479560817e4c8d in 1739ms, sequenceid=37, compaction requested=false 2023-04-08 02:54:17,283 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:17,283 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:17,283 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:17,283 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:17,283 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:17,283 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:17,283 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:17,286 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:17,286 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:17,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:17,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:17,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:17,287 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:17,287 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:17,287 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:17,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:17,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:17,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:17,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:17,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:17,289 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1128fbb9[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:17,289 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:17,289 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:17,290 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:17,290 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:17,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:17,290 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:17,290 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:17,291 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:17,291 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:17,291 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:17,292 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:17,292 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:17,292 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:17,292 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:17,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:17,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:17,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:17,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:17,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:17,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:17,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:17,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:17,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:17,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:17,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:17,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:17,296 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:17,296 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:17,296 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:17,296 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:17,296 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:17,297 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:17,297 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:17,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:17,298 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:17,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:17,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:17,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:17,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:17,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:17,299 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:17,299 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:17,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:17,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:17,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:17,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:17,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:17,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:17,300 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:17,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:17,304 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:17,304 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:17,304 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:17,304 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:17,304 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:17,304 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:17,304 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:17,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:17,304 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:17,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:17,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:17,304 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:17,304 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:17,304 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:17,304 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:17,304 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:17,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:17,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:17,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:17,405 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:17,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:17,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:17,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:17,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:17,409 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:17,409 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:17,409 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:17,409 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:17,409 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:17,409 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:17,410 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:17,410 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:17,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:17,411 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:17,411 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:17,411 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:17,411 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:17,411 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:17,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:17,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:17,416 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:17,416 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:17,416 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:17,416 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:17,416 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:17,417 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:17,422 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:17,422 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:17,422 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:17,422 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:17,422 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d38b4e710235c58cc9479560817e4c8d 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-08 02:54:17,423 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=A 2023-04-08 02:54:17,423 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:17,423 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=B 2023-04-08 02:54:17,423 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:17,423 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=C 2023-04-08 02:54:17,423 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:17,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:17,703 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:17,764 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:17,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922517763, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:17,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680922457759, max timestamp=1680922457762Num uniques 859; 2023-04-08 02:54:17,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11854, totalHeapSize=27176, min timestamp=1680922457759, max timestamp=1680922457762Num uniques 859; 2023-04-08 02:54:17,764 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-08 02:54:17,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=11854, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-08 02:54:17,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680922457759, max timestamp=1680922457762Num uniques 859; 2023-04-08 02:54:17,765 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=6870, suffix heap size=117064,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=150 2023-04-08 02:54:17,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:17,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922517781, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:17,783 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:17,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922517782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:17,784 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:17,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922517783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:17,852 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408290aaf314be044ee84cc8b2ba00de88f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408290aaf314be044ee84cc8b2ba00de88f 2023-04-08 02:54:17,867 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=50, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/7a385dc63ae64b84bf98a5d9bdac19c0 2023-04-08 02:54:17,869 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:17,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922517867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:17,887 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:17,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922517884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:17,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:17,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922517884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:17,888 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:17,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922517885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,074 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:18,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922518071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,090 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:18,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922518089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,090 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:18,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922518089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,090 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:18,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922518089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,278 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/b0c4c7ca8e164cefa8669b107dedc29f 2023-04-08 02:54:18,302 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=50 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/9d2a641bfa4e45d6b67794b45a66e464 2023-04-08 02:54:18,307 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/7a385dc63ae64b84bf98a5d9bdac19c0 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/7a385dc63ae64b84bf98a5d9bdac19c0 2023-04-08 02:54:18,312 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/7a385dc63ae64b84bf98a5d9bdac19c0, entries=150, sequenceid=50, filesize=25.1 K 2023-04-08 02:54:18,314 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/b0c4c7ca8e164cefa8669b107dedc29f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/b0c4c7ca8e164cefa8669b107dedc29f 2023-04-08 02:54:18,318 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/b0c4c7ca8e164cefa8669b107dedc29f, entries=150, sequenceid=50, filesize=11.6 K 2023-04-08 02:54:18,319 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/9d2a641bfa4e45d6b67794b45a66e464 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/9d2a641bfa4e45d6b67794b45a66e464 2023-04-08 02:54:18,323 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/9d2a641bfa4e45d6b67794b45a66e464, entries=150, sequenceid=50, filesize=11.6 K 2023-04-08 02:54:18,330 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=115.89 KB/118676 for d38b4e710235c58cc9479560817e4c8d in 908ms, sequenceid=50, compaction requested=true 2023-04-08 02:54:18,330 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:18,330 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:18,330 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:18,330 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:18,330 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,330 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:18,330 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:18,338 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,338 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:18,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:18,339 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,339 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:18,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:18,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:18,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:18,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:18,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c1dbb54[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:18,341 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:18,341 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,342 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,342 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:18,342 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:18,342 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:18,342 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,343 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:18,344 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,344 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:18,344 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:18,344 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:18,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:18,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:18,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:18,347 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:18,347 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:18,347 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:18,347 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:18,347 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:18,347 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:18,350 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,350 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:18,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:18,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,350 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:18,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:18,351 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:18,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:18,351 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:18,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,352 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:18,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,357 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:18,357 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:18,357 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:18,357 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:18,357 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,357 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:18,357 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:18,357 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:18,357 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:18,357 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:18,357 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,357 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:18,357 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:18,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:18,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:18,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:18,378 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d38b4e710235c58cc9479560817e4c8d 3/3 column families, dataSize=122.60 KB heapSize=344.62 KB 2023-04-08 02:54:18,379 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=A 2023-04-08 02:54:18,379 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:54:18,379 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=B 2023-04-08 02:54:18,379 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:54:18,379 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=C 2023-04-08 02:54:18,379 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:54:18,399 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:18,399 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922518395, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:18,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922518396, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:18,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922518397, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,400 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:18,400 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922518398, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,457 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:18,458 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:18,460 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:18,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:18,462 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:18,462 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:18,462 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:18,462 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:18,462 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:18,462 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,462 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:18,463 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:18,464 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:18,464 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:18,464 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,464 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:18,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:18,464 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:18,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:18,465 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:18,465 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:18,465 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:18,465 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:18,465 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:18,465 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:18,465 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:18,466 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:18,466 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:18,466 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:18,466 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:18,466 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:18,466 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:18,466 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:18,466 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,466 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:18,466 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:18,468 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,468 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:18,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:18,469 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,469 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:18,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:18,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:18,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:18,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:18,470 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a3a987b[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:18,470 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:18,470 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,471 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,471 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,471 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:18,472 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:18,472 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:18,472 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,472 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:18,473 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:18,473 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,473 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:18,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:18,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:18,473 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:18,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:18,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:18,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:18,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:18,475 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:18,475 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:18,475 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:18,475 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:18,476 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:18,476 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:18,477 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,477 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:18,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,477 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:18,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:18,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:18,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:18,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:18,477 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:18,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,481 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:18,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,483 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,483 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:18,483 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:18,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:18,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:18,484 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:18,484 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,484 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,484 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,484 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,484 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:18,484 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:18,484 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:18,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:18,484 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:18,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:18,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:18,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:18,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:18,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:18,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922518501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:18,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922518501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,502 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:18,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:18,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922518501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922518501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,584 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:18,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:18,586 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:18,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:18,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:18,587 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:18,587 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:18,587 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:18,588 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:18,588 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,588 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:18,589 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:18,589 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:18,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:18,589 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,589 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:18,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:18,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:18,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:18,590 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:18,590 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:18,590 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:18,590 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:18,591 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:18,591 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:18,591 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:18,591 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:18,591 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:18,591 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:18,591 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:18,591 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:18,591 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:18,591 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:18,591 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,591 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:18,591 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:18,592 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,593 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,593 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:18,593 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:18,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:18,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:18,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:18,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:18,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:18,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42e05447[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:18,594 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:18,595 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,596 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,596 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:18,596 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:18,596 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:18,596 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,596 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:18,597 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,598 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:18,598 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:18,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:18,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:18,598 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:18,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:18,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:18,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:18,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:18,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:18,600 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:18,600 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:18,600 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:18,600 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:18,600 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:18,601 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,601 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:18,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:18,601 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:18,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:18,602 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:18,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:18,602 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:18,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,603 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:18,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:18,605 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57996 deadline: 1680922518603, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,605 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:18,605 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:18,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:18,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:18,607 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:18,607 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:18,607 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:18,607 DEBUG [Thread-1000] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4154 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:54:18,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:18,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:18,607 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:18,607 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:18,607 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:18,607 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:18,608 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,608 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,608 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,608 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:18,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922518703, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:18,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922518704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:18,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922518704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,707 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:18,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922518705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,708 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:18,708 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:18,710 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:18,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:18,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:18,711 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:18,711 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:18,712 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:18,712 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:18,712 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,712 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:18,714 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,714 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:18,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:18,714 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,714 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:18,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:18,715 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:18,716 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:18,716 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:18,716 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:18,716 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:18,716 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:18,720 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:18,720 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:18,720 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:18,720 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:18,720 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:18,720 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:18,720 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:18,720 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:18,720 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,720 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:18,720 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:18,722 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,722 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:18,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:18,722 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,722 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:18,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:18,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:18,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:18,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:18,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14c39a91[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:18,724 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:18,724 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,725 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,725 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:18,725 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:18,725 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:18,725 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,725 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:18,730 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,730 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:18,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:18,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:18,730 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:18,730 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:18,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:18,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:18,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:18,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:18,732 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:18,732 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:18,732 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:18,732 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:18,732 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:18,732 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:18,735 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,735 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:18,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:18,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:18,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,738 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:18,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:18,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:18,738 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:18,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,740 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:18,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,742 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,742 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:18,742 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:18,742 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:18,742 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,742 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:18,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:18,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:18,742 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:18,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:18,742 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:18,742 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,742 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,742 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,743 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:18,743 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:18,743 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:18,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:18,795 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304082c07a0f7ef584ee2a27cacfde9b15d6a to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304082c07a0f7ef584ee2a27cacfde9b15d6a 2023-04-08 02:54:18,843 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:18,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:18,845 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:18,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:18,847 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:18,847 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:18,847 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:18,847 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:18,847 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:18,848 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,848 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:18,850 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:18,850 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:18,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:18,850 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,850 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:18,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:18,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:18,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:18,851 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:18,851 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:18,851 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:18,851 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:18,851 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:18,851 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:18,852 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:18,852 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:18,852 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:18,852 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:18,852 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:18,852 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:18,852 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:18,852 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:18,852 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,852 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:18,852 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:18,853 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:18,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:18,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:18,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:18,854 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,855 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,855 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:18,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:18,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:18,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15855446[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:18,856 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:18,856 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,857 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,857 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:18,857 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:18,857 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:18,857 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,857 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:18,858 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:18,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:18,859 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:18,859 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:18,859 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:18,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:18,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:18,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:18,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:18,862 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:18,862 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:18,862 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:18,862 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:18,862 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:18,862 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:18,865 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,865 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,865 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,865 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:18,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:18,866 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,866 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:18,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:18,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,868 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:18,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,870 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:18,870 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:18,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:18,870 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,870 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:18,870 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:18,870 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,870 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,870 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:18,870 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:18,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:18,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:18,872 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:18,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:18,872 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:18,872 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:18,872 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:18,972 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:18,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:18,975 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:18,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:18,983 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:18,983 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:18,984 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:18,984 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:18,984 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:18,984 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,984 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:18,986 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:18,986 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:18,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:18,986 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,986 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:18,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:18,987 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:18,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:18,987 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:18,992 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:18,992 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:18,992 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:18,992 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:18,992 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:18,992 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:18,992 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:18,992 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:18,992 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:18,993 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:18,993 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:18,993 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:18,993 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:18,993 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,993 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:18,993 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:18,995 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,995 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:18,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:18,995 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,995 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:18,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:18,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:18,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:18,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:18,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:18,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:18,997 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6b82efcd[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:18,997 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:18,997 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,999 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,999 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:18,999 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:18,999 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:18,999 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:19,000 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,000 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:19,001 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:19,001 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:19,001 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,001 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:19,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:19,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:19,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:19,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:19,010 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:19,010 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:19,010 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:19,010 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:19,010 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:19,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:19,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922519008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,010 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:19,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922519008, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,011 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,011 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:19,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,011 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:19,011 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922519009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:19,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:19,012 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:19,012 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,012 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922519009, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:19,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:19,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:19,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:19,014 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:19,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,017 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:19,017 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:19,017 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,017 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,017 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:19,017 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,017 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,017 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,017 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:19,017 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:19,017 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:19,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:19,017 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:19,017 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:19,017 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:19,018 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:19,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:19,118 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:19,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:19,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:19,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:19,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:19,121 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:19,121 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:19,121 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:19,121 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:19,122 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,122 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,124 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:19,124 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,125 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:19,125 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,125 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:19,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,125 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:19,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,126 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:19,126 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:19,126 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:19,126 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:19,126 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:19,126 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:19,126 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:19,126 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:19,126 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:19,126 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:19,127 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:19,127 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:19,127 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:19,127 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:19,127 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,127 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:19,127 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:19,128 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,128 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:19,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:19,128 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,129 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:19,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:19,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b80dc96[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:19,130 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:19,130 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,131 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,131 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:19,131 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:19,131 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:19,132 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,132 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:19,133 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,133 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:19,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:19,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:19,133 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:19,133 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:19,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:19,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:19,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:19,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:19,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:19,135 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:19,135 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:19,135 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:19,135 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:19,135 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:19,140 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,140 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,140 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:19,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:19,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:19,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:19,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:19,142 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:19,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:19,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,144 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:19,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,146 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,146 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:19,146 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:19,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:19,146 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,146 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:19,146 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,146 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,147 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,147 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,146 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:19,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:19,146 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:19,146 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:19,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:19,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:19,147 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:19,147 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:19,209 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=84, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/bf551fae7fab470392046441b9f229df 2023-04-08 02:54:19,247 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:19,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:19,249 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:19,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:19,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:19,252 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:19,253 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:19,253 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:19,253 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:19,253 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,253 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,254 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:19,254 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:19,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:19,254 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,255 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,255 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:19,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,255 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:19,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,256 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:19,256 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:19,256 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:19,256 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:19,256 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:19,256 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:19,256 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:19,256 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:19,256 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:19,256 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:19,257 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:19,257 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:19,257 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:19,257 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:19,257 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,257 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:19,257 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:19,259 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,259 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:19,260 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,260 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:19,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:19,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:19,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:19,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:19,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@231fcc65[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:19,262 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:19,262 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,263 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,263 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:19,263 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:19,263 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,263 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:19,263 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:19,264 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,264 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:19,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:19,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:19,264 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:19,265 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:19,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:19,267 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:19,267 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:19,267 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:19,267 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:19,267 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:19,268 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,268 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,268 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:19,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:19,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:19,269 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:19,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,270 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,273 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,273 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:19,273 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,273 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,273 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,273 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:19,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:19,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:19,273 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:19,274 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,274 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:19,274 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:19,274 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:19,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:19,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:19,274 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:19,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:19,374 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:19,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:19,377 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:19,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:19,380 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:19,380 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:19,380 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:19,380 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:19,380 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:19,381 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,381 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,382 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:19,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:19,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:19,382 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,382 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,383 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:19,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:19,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:19,388 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:19,388 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:19,388 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:19,388 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:19,388 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:19,389 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:19,389 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:19,389 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:19,389 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:19,389 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:19,389 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:19,389 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:19,389 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:19,389 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,389 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:19,389 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:19,391 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:19,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:19,391 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:19,392 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,392 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:19,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:19,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:19,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:19,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c078b24[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:19,395 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:19,395 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,396 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,396 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:19,396 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:19,396 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:19,397 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,397 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:19,399 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:19,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:19,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:19,401 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:19,401 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:19,401 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:19,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:19,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:19,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:19,409 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:19,409 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:19,409 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:19,409 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:19,409 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:19,409 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:19,412 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,412 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:19,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:19,412 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:19,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:19,413 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,413 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:19,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:19,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:19,417 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,417 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:19,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,421 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:19,421 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:19,421 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:19,421 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:19,421 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,421 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,421 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,421 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,421 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,421 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,421 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,421 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:19,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:19,421 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:19,421 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:19,421 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:19,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:19,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:19,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:19,513 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:19,514 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922519511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:19,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922519514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,517 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:19,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922519515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,520 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:19,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922519517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,522 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:19,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:19,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:19,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:19,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:19,525 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:19,525 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:19,525 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:19,525 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:19,526 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,526 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,528 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:19,528 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:19,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:19,529 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,529 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:19,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:19,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,529 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:19,530 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:19,530 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:19,530 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:19,530 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:19,530 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:19,532 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:19,532 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:19,532 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:19,532 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:19,532 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:19,532 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:19,532 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:19,532 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:19,532 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,532 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:19,532 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:19,534 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,534 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:19,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:19,535 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,535 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:19,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:19,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:19,538 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6007f3ee[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:19,538 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:19,538 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,544 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,544 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,544 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,544 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,544 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:19,545 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:19,545 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:19,545 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:19,549 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,549 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:19,549 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:19,549 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:19,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:19,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:19,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:19,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:19,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:19,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:19,552 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:19,552 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:19,552 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:19,552 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:19,552 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:19,552 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:19,554 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,554 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,554 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:19,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:19,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:19,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:19,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:19,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:19,555 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:19,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,557 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:19,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,561 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,562 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,562 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:19,562 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:19,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:19,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:19,562 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:19,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:19,562 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:19,562 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:19,562 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,562 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,562 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,562 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,562 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:19,562 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:19,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:19,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:19,626 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/d095085398d640208fff3f508aa78d31 2023-04-08 02:54:19,662 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:19,663 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:19,665 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:19,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:19,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:19,672 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:19,672 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:19,672 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:19,672 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:19,673 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,673 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,674 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:19,674 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,674 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:19,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:19,674 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,674 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:19,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,675 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:19,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,675 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:19,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:19,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:19,676 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:19,676 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:19,676 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:19,676 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:19,676 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:19,676 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:19,676 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:19,677 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:19,677 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:19,677 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:19,677 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:19,677 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,677 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:19,677 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:19,679 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,680 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:19,680 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,680 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:19,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:19,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:19,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:19,682 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4dcee714[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:19,682 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:19,682 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,683 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,684 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:19,684 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:19,684 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:19,684 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,684 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:19,685 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,685 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:19,686 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:19,686 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:19,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:19,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:19,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:19,689 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:19,689 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:19,689 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:19,689 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:19,689 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:19,689 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:19,691 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,691 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,691 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:19,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:19,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:19,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:19,692 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,693 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:19,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,697 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,697 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,697 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,697 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,697 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,697 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,697 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,697 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:19,697 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:19,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:19,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:19,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:19,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:19,697 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:19,697 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:19,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:19,698 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:19,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:19,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:19,798 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:19,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:19,800 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:19,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:19,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:19,801 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:19,801 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:19,801 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:19,801 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:19,802 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,802 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,803 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:19,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:19,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:19,803 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,804 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,804 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:19,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:19,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:19,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:19,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:19,805 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:19,805 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:19,805 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:19,805 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:19,805 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:19,805 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:19,805 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:19,806 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:19,806 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:19,806 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:19,806 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:19,806 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,806 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:19,806 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:19,807 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,807 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:19,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:19,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:19,808 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,808 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:19,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:19,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:19,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:19,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@106f762c[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:19,809 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:19,809 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,811 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,812 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:19,812 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:19,812 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:19,812 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,812 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:19,813 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:19,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:19,813 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:19,813 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:19,813 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:19,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:19,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:19,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:19,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:19,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:19,815 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:19,815 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:19,815 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:19,815 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:19,815 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:19,817 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,817 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:19,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,817 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:19,817 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:19,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:19,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,819 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,821 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,821 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,821 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,821 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:19,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:19,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:19,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:19,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:19,821 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:19,821 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:19,821 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:19,821 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:19,821 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:19,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:19,822 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,822 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,822 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:19,922 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:19,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:19,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:19,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:19,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:19,925 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:19,925 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:19,925 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:19,925 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:19,926 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,926 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,928 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:19,928 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:19,928 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,928 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:19,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:19,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,929 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:19,929 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:19,930 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:19,930 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:19,930 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:19,930 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:19,930 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:19,930 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:19,930 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:19,930 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:19,930 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:19,930 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:19,930 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:19,930 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:19,930 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,930 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:19,930 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:19,932 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,932 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:19,932 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,932 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:19,932 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:19,933 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:19,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@301191ae[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:19,934 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:19,934 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,935 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,935 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,935 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:19,935 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:19,935 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,935 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:19,935 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:19,936 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:19,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:19,936 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:19,936 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:19,936 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:19,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:19,938 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:19,938 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:19,938 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:19,938 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:19,938 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:19,938 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:19,941 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,941 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,941 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:19,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:19,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:19,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,942 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:19,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,943 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:19,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:19,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,947 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,947 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:19,947 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:19,947 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:19,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:19,947 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:19,947 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:19,948 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:19,948 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:19,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:19,947 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:19,948 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:19,948 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:19,948 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,948 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:19,948 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:19,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:19,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:20,042 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=37.62 KB at sequenceid=84 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/a3a95cfe2de949ccba1f9d5a3b00ea71 2023-04-08 02:54:20,048 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/bf551fae7fab470392046441b9f229df as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/bf551fae7fab470392046441b9f229df 2023-04-08 02:54:20,048 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:20,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:20,053 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/bf551fae7fab470392046441b9f229df, entries=200, sequenceid=84, filesize=31.9 K 2023-04-08 02:54:20,054 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:20,054 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/d095085398d640208fff3f508aa78d31 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/d095085398d640208fff3f508aa78d31 2023-04-08 02:54:20,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:20,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:20,055 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:20,055 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:20,055 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:20,055 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:20,056 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:20,056 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:20,057 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:20,057 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:20,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:20,058 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,058 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:20,058 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:20,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:20,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:20,059 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/d095085398d640208fff3f508aa78d31, entries=150, sequenceid=84, filesize=11.6 K 2023-04-08 02:54:20,059 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:20,059 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:20,059 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:20,059 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:20,059 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:20,059 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:20,059 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:20,059 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:20,059 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:20,059 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:20,059 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/a3a95cfe2de949ccba1f9d5a3b00ea71 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a3a95cfe2de949ccba1f9d5a3b00ea71 2023-04-08 02:54:20,059 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:20,059 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:20,060 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:20,060 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:20,060 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,060 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:20,060 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:20,061 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,061 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:20,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:20,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:20,062 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:20,062 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:20,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:20,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:20,064 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a3a95cfe2de949ccba1f9d5a3b00ea71, entries=150, sequenceid=84, filesize=11.6 K 2023-04-08 02:54:20,065 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~122.60 KB/125546, heapSize ~344.57 KB/352840, currentSize=60.38 KB/61830 for d38b4e710235c58cc9479560817e4c8d in 1687ms, sequenceid=84, compaction requested=true 2023-04-08 02:54:20,065 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:20,065 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:20,065 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:20,065 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:20,065 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:54:20,065 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:54:20,066 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:54:20,067 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d38b4e710235c58cc9479560817e4c8d/B is initiating minor compaction (all files) 2023-04-08 02:54:20,067 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 123808 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:54:20,067 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d38b4e710235c58cc9479560817e4c8d/B in TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:20,067 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): d38b4e710235c58cc9479560817e4c8d/A is initiating minor compaction (all files) 2023-04-08 02:54:20,067 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/72811cc7722144bab7f684d1cad2be1b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/97d505898f1f442c972b27c3a7badb61, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/b0c4c7ca8e164cefa8669b107dedc29f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/d095085398d640208fff3f508aa78d31] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp, totalSize=46.4 K 2023-04-08 02:54:20,067 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d38b4e710235c58cc9479560817e4c8d/A in TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:20,067 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/080a10550c5a417e880d276fa08a8dbe, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/c6f6445e4bf14b2f9b7e03c4c4bbf616, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/7a385dc63ae64b84bf98a5d9bdac19c0, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/bf551fae7fab470392046441b9f229df] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp, totalSize=120.9 K 2023-04-08 02:54:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,067 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 72811cc7722144bab7f684d1cad2be1b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1680922454402 2023-04-08 02:54:20,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:20,067 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 080a10550c5a417e880d276fa08a8dbe, keycount=250, bloomtype=ROW, size=38.7 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1680922454402 2023-04-08 02:54:20,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:20,068 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 97d505898f1f442c972b27c3a7badb61, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680922454449 2023-04-08 02:54:20,068 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f569cc1[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:20,068 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:20,068 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:20,068 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting c6f6445e4bf14b2f9b7e03c4c4bbf616, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680922454449 2023-04-08 02:54:20,069 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting b0c4c7ca8e164cefa8669b107dedc29f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680922455568 2023-04-08 02:54:20,069 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting d095085398d640208fff3f508aa78d31, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680922457781 2023-04-08 02:54:20,070 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:20,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:20,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:20,070 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:20,070 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,070 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:20,070 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:20,070 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:20,076 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 7a385dc63ae64b84bf98a5d9bdac19c0, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680922455568 2023-04-08 02:54:20,077 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting bf551fae7fab470392046441b9f229df, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680922457770 2023-04-08 02:54:20,087 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,087 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:20,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:20,087 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:20,088 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:20,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:20,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:20,090 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:20,090 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:20,090 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:20,090 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:20,090 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:20,090 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:20,092 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:20,092 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:20,092 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:20,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:20,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:20,092 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:20,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:20,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:20,094 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:20,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:20,096 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:20,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,099 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:20,099 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:20,099 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:20,099 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:20,099 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:20,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:20,099 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:20,099 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:20,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:20,099 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:20,099 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,099 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:20,099 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:20,099 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:20,099 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:20,100 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:20,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:20,100 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:20,114 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d38b4e710235c58cc9479560817e4c8d#B#compaction#147 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:20,126 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): d38b4e710235c58cc9479560817e4c8d#A#compaction#146 average throughput is 0.49 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:20,138 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/4668bb62f81d4421ac705a03356486a9 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/4668bb62f81d4421ac705a03356486a9 2023-04-08 02:54:20,149 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d38b4e710235c58cc9479560817e4c8d/B of d38b4e710235c58cc9479560817e4c8d into 4668bb62f81d4421ac705a03356486a9(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:20,149 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:20,149 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d., storeName=d38b4e710235c58cc9479560817e4c8d/B, priority=12, startTime=1680922460065; duration=0sec 2023-04-08 02:54:20,149 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:20,149 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:54:20,153 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:54:20,153 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d38b4e710235c58cc9479560817e4c8d/C is initiating minor compaction (all files) 2023-04-08 02:54:20,153 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d38b4e710235c58cc9479560817e4c8d/C in TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:20,153 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/0ff8bedb3705457fbe57b88601b98914, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/0bc82dc108204b4ba933b984799b2a92, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/9d2a641bfa4e45d6b67794b45a66e464, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a3a95cfe2de949ccba1f9d5a3b00ea71] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp, totalSize=46.4 K 2023-04-08 02:54:20,153 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 0ff8bedb3705457fbe57b88601b98914, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=18, earliestPutTs=1680922454402 2023-04-08 02:54:20,154 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 0bc82dc108204b4ba933b984799b2a92, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=37, earliestPutTs=1680922454449 2023-04-08 02:54:20,154 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 9d2a641bfa4e45d6b67794b45a66e464, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=50, earliestPutTs=1680922455568 2023-04-08 02:54:20,155 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting a3a95cfe2de949ccba1f9d5a3b00ea71, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680922457781 2023-04-08 02:54:20,168 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d38b4e710235c58cc9479560817e4c8d#C#compaction#148 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:20,198 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/1c999831d17c498b93098e5b4aa8c229 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/1c999831d17c498b93098e5b4aa8c229 2023-04-08 02:54:20,200 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:20,205 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d38b4e710235c58cc9479560817e4c8d/C of d38b4e710235c58cc9479560817e4c8d into 1c999831d17c498b93098e5b4aa8c229(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:20,205 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:20,205 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d., storeName=d38b4e710235c58cc9479560817e4c8d/C, priority=12, startTime=1680922460065; duration=0sec 2023-04-08 02:54:20,205 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:20,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:20,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:20,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:20,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:20,215 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:20,215 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:20,215 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:20,215 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:20,218 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:20,218 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:20,219 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:20,219 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:20,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:20,220 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,220 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:20,220 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:20,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:20,221 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:20,221 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:20,221 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:20,221 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:20,221 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:20,236 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:20,236 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:20,237 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:20,238 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:20,238 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d38b4e710235c58cc9479560817e4c8d 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-08 02:54:20,239 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=A 2023-04-08 02:54:20,239 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:20,239 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=B 2023-04-08 02:54:20,239 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:20,239 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=C 2023-04-08 02:54:20,239 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:20,263 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304087e71adece28844718ebe22b519492f8b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304087e71adece28844718ebe22b519492f8b 2023-04-08 02:54:20,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:20,522 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:20,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:20,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922520546, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1680922460529, max timestamp=1680922460548Num uniques 859; 2023-04-08 02:54:20,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-08 02:54:20,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1680922460529, max timestamp=1680922460548Num uniques 859; 2023-04-08 02:54:20,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-08 02:54:20,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14144, totalHeapSize=32376, min timestamp=1680922460529, max timestamp=1680922460548Num uniques 859; 2023-04-08 02:54:20,551 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=14144, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-08 02:54:20,571 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:20,571 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:20,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922520568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922520569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,571 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:20,571 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922520569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,652 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:20,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922520652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,669 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=98, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/c25e5fecc86b45c49ca41cba83c61cab 2023-04-08 02:54:20,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:20,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:20,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922520673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922520673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:20,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922520673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,854 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:20,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922520854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:20,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922520875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:20,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922520875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:20,876 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:20,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922520875, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,081 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/3559e3e1c5114d78b7e4bca509d062db 2023-04-08 02:54:21,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:21,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922521156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,179 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:21,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922521177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,182 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:21,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922521178, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,182 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:21,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922521179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,353 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/b872522a3af647b4aa4d477f5b7cb4f4 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/b872522a3af647b4aa4d477f5b7cb4f4 2023-04-08 02:54:21,359 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d38b4e710235c58cc9479560817e4c8d/A of d38b4e710235c58cc9479560817e4c8d into b872522a3af647b4aa4d477f5b7cb4f4(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-08 02:54:21,359 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:21,359 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d., storeName=d38b4e710235c58cc9479560817e4c8d/A, priority=12, startTime=1680922460065; duration=1sec 2023-04-08 02:54:21,359 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:21,498 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=98 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/521c0c2e2ded4588ae29ab832594555a 2023-04-08 02:54:21,503 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/c25e5fecc86b45c49ca41cba83c61cab as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/c25e5fecc86b45c49ca41cba83c61cab 2023-04-08 02:54:21,507 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/c25e5fecc86b45c49ca41cba83c61cab, entries=150, sequenceid=98, filesize=25.1 K 2023-04-08 02:54:21,508 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/3559e3e1c5114d78b7e4bca509d062db as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/3559e3e1c5114d78b7e4bca509d062db 2023-04-08 02:54:21,512 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/3559e3e1c5114d78b7e4bca509d062db, entries=150, sequenceid=98, filesize=11.6 K 2023-04-08 02:54:21,513 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/521c0c2e2ded4588ae29ab832594555a as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/521c0c2e2ded4588ae29ab832594555a 2023-04-08 02:54:21,517 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/521c0c2e2ded4588ae29ab832594555a, entries=150, sequenceid=98, filesize=11.6 K 2023-04-08 02:54:21,518 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for d38b4e710235c58cc9479560817e4c8d in 1280ms, sequenceid=98, compaction requested=false 2023-04-08 02:54:21,518 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:21,518 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:21,518 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:21,518 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:21,518 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,518 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:21,518 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:21,521 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,521 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:21,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:21,521 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:21,521 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:21,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:21,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@559a2078[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:21,523 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:21,523 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:21,524 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:21,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:21,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:21,524 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:21,524 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:21,524 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:21,524 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,525 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:21,525 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:21,525 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:21,525 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,525 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:21,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:21,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:21,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:21,528 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:21,528 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:21,528 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:21,528 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:21,528 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:21,530 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:21,530 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:21,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:21,530 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:21,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:21,531 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:21,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:21,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:21,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,532 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:21,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:21,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,536 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:21,536 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:21,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:21,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:21,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:21,536 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:21,536 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,536 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:21,536 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:21,536 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,536 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:21,536 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:21,536 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:21,536 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:21,536 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:21,536 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:21,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:21,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:21,636 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:21,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:21,639 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:21,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:21,640 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:21,640 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:21,640 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:21,640 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:21,640 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:21,640 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:21,641 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:21,643 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:21,643 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:21,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:21,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:21,644 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,644 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:21,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:21,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:21,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:21,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:21,645 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:21,645 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:21,645 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:21,645 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:21,646 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:21,646 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:21,646 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:21,646 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:21,646 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:21,646 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d38b4e710235c58cc9479560817e4c8d 3/3 column families, dataSize=127.47 KB heapSize=351 KB 2023-04-08 02:54:21,646 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=A 2023-04-08 02:54:21,646 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:54:21,646 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=B 2023-04-08 02:54:21,647 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:54:21,647 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=C 2023-04-08 02:54:21,647 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:54:21,663 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408a03c3cc5830c4dedb349a6c669590b16 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408a03c3cc5830c4dedb349a6c669590b16 2023-04-08 02:54:21,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:21,664 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:21,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:21,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922521686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:21,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922521686, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:21,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922521687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,690 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:21,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922521687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,791 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:21,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922521788, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,792 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:21,792 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922521789, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,794 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:21,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922521791, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,994 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:21,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922521993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:21,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922521993, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:21,997 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:21,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922521996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,070 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=132, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/3b24d5cc7faf403f89d3af951aabc72c 2023-04-08 02:54:22,298 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:22,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922522296, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,299 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:22,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922522298, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,302 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:22,302 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922522300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,483 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/eb548040fef54907bd18a9e0eaef4424 2023-04-08 02:54:22,494 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=132 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/794438b911d54527b31701c92d7cec5b 2023-04-08 02:54:22,499 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/3b24d5cc7faf403f89d3af951aabc72c as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/3b24d5cc7faf403f89d3af951aabc72c 2023-04-08 02:54:22,503 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/3b24d5cc7faf403f89d3af951aabc72c, entries=150, sequenceid=132, filesize=25.2 K 2023-04-08 02:54:22,504 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/eb548040fef54907bd18a9e0eaef4424 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/eb548040fef54907bd18a9e0eaef4424 2023-04-08 02:54:22,508 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/eb548040fef54907bd18a9e0eaef4424, entries=150, sequenceid=132, filesize=11.7 K 2023-04-08 02:54:22,509 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/794438b911d54527b31701c92d7cec5b as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/794438b911d54527b31701c92d7cec5b 2023-04-08 02:54:22,513 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/794438b911d54527b31701c92d7cec5b, entries=150, sequenceid=132, filesize=11.7 K 2023-04-08 02:54:22,514 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~350.95 KB/359376, currentSize=67.09 KB/68700 for d38b4e710235c58cc9479560817e4c8d in 868ms, sequenceid=132, compaction requested=true 2023-04-08 02:54:22,514 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:22,514 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:22,514 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:22,514 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:22,514 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,514 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:22,514 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:22,521 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,521 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:22,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:22,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:22,521 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,521 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:22,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:22,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:22,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:22,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:22,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3eafe2f1[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:22,523 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:22,523 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,526 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,526 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,527 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:22,527 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,527 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:22,527 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:22,527 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:22,528 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:22,528 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,528 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:22,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:22,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:22,528 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:22,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:22,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:22,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:22,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:22,530 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:22,530 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:22,531 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:22,531 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:22,531 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:22,531 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:22,533 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,533 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:22,533 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,533 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:22,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:22,534 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:22,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,535 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:22,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,538 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,538 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:22,538 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:22,538 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,538 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:22,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:22,538 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:22,538 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,539 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,539 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,538 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:22,539 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:22,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:22,539 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:22,539 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:22,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:22,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:22,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:22,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:22,633 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d38b4e710235c58cc9479560817e4c8d 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-08 02:54:22,633 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=A 2023-04-08 02:54:22,634 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:22,634 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=B 2023-04-08 02:54:22,634 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:22,634 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=C 2023-04-08 02:54:22,634 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:22,639 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:22,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:22,641 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:22,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:22,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:22,643 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:22,643 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:22,643 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:22,643 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:22,643 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,643 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:22,645 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,645 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:22,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:22,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:22,646 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,646 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:22,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:22,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:22,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:22,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:22,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:22,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:22,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:22,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:22,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:22,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:22,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:22,648 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:22,648 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:22,648 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:22,648 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:22,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:22,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:22,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:22,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:22,650 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408979daf36b0b148108e4872c00c64f912 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408979daf36b0b148108e4872c00c64f912 2023-04-08 02:54:22,652 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,652 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:22,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:22,653 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,653 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:22,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:22,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:22,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:22,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:22,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:22,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@20f886b[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:22,654 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:22,654 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,656 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,656 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:22,656 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:22,656 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:22,657 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,657 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:22,660 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,660 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:22,660 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:22,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:22,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:22,660 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:22,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:22,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:22,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:22,663 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:22,663 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:22,663 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:22,663 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:22,663 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:22,663 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:22,664 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,664 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:22,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,664 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:22,665 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:22,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:22,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:22,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:22,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:22,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,667 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,669 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,669 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:22,669 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:22,669 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:22,669 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:22,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:22,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:22,669 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:22,669 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:22,669 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:22,670 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,670 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,670 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,670 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,670 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:22,670 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:22,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:22,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:22,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:22,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57996 deadline: 1680922522677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,696 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:22,696 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922522695, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,770 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:22,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:22,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:22,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:22,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:22,773 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:22,773 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:22,773 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:22,773 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:22,774 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,774 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:22,775 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:22,775 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:22,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:22,775 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,775 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:22,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:22,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:22,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:22,776 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:22,776 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:22,776 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:22,776 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:22,777 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:22,777 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:22,777 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:22,777 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:22,777 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:22,777 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:22,777 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:22,777 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:22,777 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:22,777 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:22,777 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,777 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:22,777 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:22,779 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,779 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:22,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:22,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:22,780 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,780 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:22,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:22,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:22,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:22,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:22,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5964bc7f[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:22,781 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:22,782 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,782 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:22,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57996 deadline: 1680922522779, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,783 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,783 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,783 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:22,783 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:22,783 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:22,783 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:22,786 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,786 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:22,786 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:22,786 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:22,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:22,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:22,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:22,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:22,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:22,789 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:22,789 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:22,789 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:22,789 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:22,789 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:22,789 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:22,790 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,790 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:22,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:22,790 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:22,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:22,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:22,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:22,791 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:22,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:22,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:22,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,792 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:22,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:22,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,794 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,794 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:22,794 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:22,794 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:22,794 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:22,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:22,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:22,794 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:22,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:22,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:22,794 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:22,794 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,794 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,794 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,794 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:22,794 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:22,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:22,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:22,799 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:22,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922522799, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:22,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922522804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,805 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:22,805 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922522804, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,895 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:22,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:22,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:22,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:22,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:22,898 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:22,898 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:22,898 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:22,898 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:22,899 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,899 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:22,900 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:22,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:22,900 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:22,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:22,901 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,901 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:22,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:22,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:22,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:22,902 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:22,902 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:22,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:22,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:22,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:22,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:22,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:22,902 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:22,902 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:22,902 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:22,902 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:22,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:22,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:22,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:22,903 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:22,905 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:22,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:22,906 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:22,906 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,906 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:22,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:22,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:22,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:22,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:22,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5d79cecb[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:22,909 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:22,909 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,911 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,911 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:22,911 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:22,911 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:22,911 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,911 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:22,912 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,912 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:22,912 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:22,912 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:22,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:22,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:22,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:22,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:22,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:22,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:22,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:22,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:22,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:22,915 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:22,915 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:22,915 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:22,915 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:22,915 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:22,918 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,918 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:22,918 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:22,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:22,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:22,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:22,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:22,918 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:22,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:22,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:22,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,920 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:22,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:22,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,923 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,923 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:22,923 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:22,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:22,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:22,923 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:22,923 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:22,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:22,923 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:22,923 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:22,923 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:22,923 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:22,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:22,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:22,923 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,923 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:22,923 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:22,987 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:22,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57996 deadline: 1680922522985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,023 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:23,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:23,027 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:23,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:23,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:23,028 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:23,028 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:23,028 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:23,028 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:23,029 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,029 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:23,030 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:23,030 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:23,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:23,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:23,031 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,031 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:23,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:23,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:23,031 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:23,031 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:23,032 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:23,032 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:23,032 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:23,032 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:23,032 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:23,032 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:23,032 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:23,032 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:23,032 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:23,032 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:23,032 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:23,032 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:23,032 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,032 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:23,032 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:23,034 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,034 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:23,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:23,034 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,034 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:23,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:23,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:23,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:23,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:23,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:23,036 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c0007ff[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:23,036 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:23,036 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,040 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,041 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:23,041 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:23,041 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:23,041 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,041 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:23,042 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:23,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:23,042 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:23,042 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:23,042 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:23,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:23,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:23,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:23,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:23,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:23,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:23,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:23,047 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:23,047 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:23,047 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:23,047 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:23,048 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:23,050 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,050 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,050 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:23,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:23,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:23,050 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:23,051 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:23,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:23,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,057 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=146, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/720e931a46dd4c39abcad436db3eb05a 2023-04-08 02:54:23,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:23,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:23,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:23,061 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:23,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:23,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,066 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,066 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:23,066 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:23,066 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:23,066 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:23,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:23,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:23,066 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:23,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:23,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:23,066 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,067 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,067 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,067 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,067 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:23,067 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:23,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:23,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:23,068 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:23,089 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/0d3ce136a3b245f181728b3cdb6223c9 2023-04-08 02:54:23,167 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:23,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:23,171 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:23,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:23,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:23,172 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:23,172 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:23,172 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:23,172 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:23,173 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,173 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:23,175 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:23,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:23,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:23,175 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:23,176 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,176 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:23,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:23,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:23,177 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:23,178 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:23,178 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:23,178 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:23,178 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:23,178 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:23,184 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:23,184 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:23,195 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:23,195 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:23,196 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:23,196 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:23,196 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:23,196 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:23,196 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,196 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:23,196 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:23,197 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,198 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:23,198 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,198 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:23,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:23,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:23,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:23,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:23,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62b298c4[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:23,199 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:23,199 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,200 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,201 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:23,201 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:23,201 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:23,201 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,201 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:23,202 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:23,202 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:23,202 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,202 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:23,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:23,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:23,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:23,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:23,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:23,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:23,205 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:23,205 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:23,205 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:23,205 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:23,205 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:23,208 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:23,208 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:23,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,208 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:23,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:23,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:23,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:23,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,211 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:23,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:23,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:23,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,217 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:23,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,220 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,220 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:23,220 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:23,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:23,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:23,220 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:23,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:23,220 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:23,220 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:23,220 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:23,220 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:23,220 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:23,220 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,220 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,220 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,221 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:23,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:23,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:23,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57996 deadline: 1680922523288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,320 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:23,322 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:23,325 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:23,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:23,326 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:23,326 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:23,326 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:23,326 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:23,326 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:23,327 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,327 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:23,328 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,328 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:23,328 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:23,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:23,328 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,328 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:23,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:23,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:23,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:23,329 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:23,329 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:23,329 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:23,329 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:23,329 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:23,329 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:23,329 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:23,329 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:23,329 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:23,330 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:23,330 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:23,330 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:23,330 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:23,330 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:23,330 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,330 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:23,330 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:23,331 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,331 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:23,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:23,331 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,332 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:23,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:23,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:23,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:23,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:23,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:23,333 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e4947a9[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:23,333 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:23,333 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,335 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,335 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:23,335 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:23,335 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:23,335 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,335 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:23,337 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,337 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:23,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:23,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:23,337 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:23,337 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:23,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:23,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:23,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:23,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:23,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:23,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:23,339 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:23,339 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:23,339 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:23,339 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:23,339 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:23,339 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:23,340 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,340 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:23,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:23,340 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:23,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:23,341 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:23,341 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:23,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:23,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,343 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:23,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:23,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,345 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:23,345 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:23,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:23,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:23,345 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:23,345 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:23,345 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:23,345 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:23,345 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:23,345 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,345 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,345 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,345 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,345 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:23,345 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:23,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:23,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:23,446 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:23,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:23,448 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:23,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:23,449 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:23,449 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:23,449 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:23,449 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:23,449 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:23,449 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,449 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:23,455 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,455 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:23,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:23,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:23,455 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,455 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:23,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:23,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:23,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:23,456 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:23,457 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:23,457 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:23,457 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:23,457 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:23,457 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:23,457 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:23,457 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:23,457 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:23,457 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:23,457 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:23,457 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:23,457 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:23,457 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:23,457 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,457 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:23,457 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:23,459 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,459 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:23,460 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,460 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:23,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:23,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:23,461 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d55cd3d[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:23,461 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:23,461 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,462 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,463 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:23,463 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:23,463 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:23,463 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,463 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:23,464 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,464 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:23,464 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:23,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:23,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:23,464 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:23,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:23,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:23,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:23,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:23,468 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:23,468 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:23,468 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:23,468 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:23,468 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:23,468 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:23,473 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,473 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,473 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:23,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:23,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:23,474 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,474 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:23,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:23,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:23,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:23,475 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:23,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:23,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:23,483 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:23,483 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:23,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:23,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:23,483 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,483 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:23,483 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:23,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:23,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:23,484 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:23,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:23,484 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,484 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,484 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:23,484 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,484 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:23,484 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:23,485 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:23,485 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:23,504 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=146 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/50fd46b457b24105b9b975fb761a8d9c 2023-04-08 02:54:23,509 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/720e931a46dd4c39abcad436db3eb05a as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/720e931a46dd4c39abcad436db3eb05a 2023-04-08 02:54:23,514 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/720e931a46dd4c39abcad436db3eb05a, entries=150, sequenceid=146, filesize=25.3 K 2023-04-08 02:54:23,514 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/0d3ce136a3b245f181728b3cdb6223c9 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/0d3ce136a3b245f181728b3cdb6223c9 2023-04-08 02:54:23,519 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/0d3ce136a3b245f181728b3cdb6223c9, entries=150, sequenceid=146, filesize=11.8 K 2023-04-08 02:54:23,520 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/50fd46b457b24105b9b975fb761a8d9c as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/50fd46b457b24105b9b975fb761a8d9c 2023-04-08 02:54:23,532 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/50fd46b457b24105b9b975fb761a8d9c, entries=150, sequenceid=146, filesize=11.8 K 2023-04-08 02:54:23,533 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for d38b4e710235c58cc9479560817e4c8d in 900ms, sequenceid=146, compaction requested=true 2023-04-08 02:54:23,533 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:23,533 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:23,533 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:23,533 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:54:23,533 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:54:23,533 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:23,534 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47932 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:54:23,534 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 103372 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:54:23,534 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d38b4e710235c58cc9479560817e4c8d/B is initiating minor compaction (all files) 2023-04-08 02:54:23,534 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): d38b4e710235c58cc9479560817e4c8d/A is initiating minor compaction (all files) 2023-04-08 02:54:23,535 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d38b4e710235c58cc9479560817e4c8d/B in TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:23,535 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d38b4e710235c58cc9479560817e4c8d/A in TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:23,535 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/4668bb62f81d4421ac705a03356486a9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/3559e3e1c5114d78b7e4bca509d062db, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/eb548040fef54907bd18a9e0eaef4424, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/0d3ce136a3b245f181728b3cdb6223c9] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp, totalSize=46.8 K 2023-04-08 02:54:23,535 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/b872522a3af647b4aa4d477f5b7cb4f4, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/c25e5fecc86b45c49ca41cba83c61cab, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/3b24d5cc7faf403f89d3af951aabc72c, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/720e931a46dd4c39abcad436db3eb05a] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp, totalSize=100.9 K 2023-04-08 02:54:23,535 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 4668bb62f81d4421ac705a03356486a9, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680922457781 2023-04-08 02:54:23,535 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting b872522a3af647b4aa4d477f5b7cb4f4, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680922457781 2023-04-08 02:54:23,535 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 3559e3e1c5114d78b7e4bca509d062db, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680922458395 2023-04-08 02:54:23,535 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting c25e5fecc86b45c49ca41cba83c61cab, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680922458395 2023-04-08 02:54:23,536 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting eb548040fef54907bd18a9e0eaef4424, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1680922460560 2023-04-08 02:54:23,536 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 3b24d5cc7faf403f89d3af951aabc72c, keycount=150, bloomtype=ROW, size=25.2 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1680922460560 2023-04-08 02:54:23,536 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 0d3ce136a3b245f181728b3cdb6223c9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1680922461685 2023-04-08 02:54:23,536 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 720e931a46dd4c39abcad436db3eb05a, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1680922461685 2023-04-08 02:54:23,549 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d38b4e710235c58cc9479560817e4c8d#B#compaction#158 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:23,558 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): d38b4e710235c58cc9479560817e4c8d#A#compaction#159 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:23,564 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/66855a5eb08041cebf8d309d1bace3ef as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/66855a5eb08041cebf8d309d1bace3ef 2023-04-08 02:54:23,569 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d38b4e710235c58cc9479560817e4c8d/B of d38b4e710235c58cc9479560817e4c8d into 66855a5eb08041cebf8d309d1bace3ef(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:23,569 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:23,569 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d., storeName=d38b4e710235c58cc9479560817e4c8d/B, priority=12, startTime=1680922463533; duration=0sec 2023-04-08 02:54:23,569 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:23,569 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:54:23,571 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47932 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:54:23,571 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d38b4e710235c58cc9479560817e4c8d/C is initiating minor compaction (all files) 2023-04-08 02:54:23,571 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d38b4e710235c58cc9479560817e4c8d/C in TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:23,571 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/1c999831d17c498b93098e5b4aa8c229, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/521c0c2e2ded4588ae29ab832594555a, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/794438b911d54527b31701c92d7cec5b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/50fd46b457b24105b9b975fb761a8d9c] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp, totalSize=46.8 K 2023-04-08 02:54:23,571 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 1c999831d17c498b93098e5b4aa8c229, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=84, earliestPutTs=1680922457781 2023-04-08 02:54:23,572 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 521c0c2e2ded4588ae29ab832594555a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=98, earliestPutTs=1680922458395 2023-04-08 02:54:23,572 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 794438b911d54527b31701c92d7cec5b, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=132, earliestPutTs=1680922460560 2023-04-08 02:54:23,572 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 50fd46b457b24105b9b975fb761a8d9c, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1680922461685 2023-04-08 02:54:23,580 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d38b4e710235c58cc9479560817e4c8d#C#compaction#160 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:23,585 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:23,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:23,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:23,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:23,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:23,588 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:23,588 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:23,588 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:23,588 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:23,589 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,589 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:23,590 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,590 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:23,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:23,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:23,590 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,590 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:23,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:23,592 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:23,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:23,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:23,593 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:23,593 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:23,593 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:23,593 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:23,593 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:23,593 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:23,593 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:23,593 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:23,593 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:23,593 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:23,594 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d38b4e710235c58cc9479560817e4c8d 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-08 02:54:23,594 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=A 2023-04-08 02:54:23,594 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:23,594 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=B 2023-04-08 02:54:23,594 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:23,594 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=C 2023-04-08 02:54:23,594 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:23,611 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304089d56cb3c86b2419ca74d66abf849103e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304089d56cb3c86b2419ca74d66abf849103e 2023-04-08 02:54:23,634 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=165, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/e9f8c45ddb6449a38207a6d70e666332 2023-04-08 02:54:23,794 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:23,794 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:23,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:23,817 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922523815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,817 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:23,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922523815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:23,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922523816, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,818 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:23,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57996 deadline: 1680922523817, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,922 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:23,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922523919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:23,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57996 deadline: 1680922523919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:23,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922523920, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:23,983 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/2f944dee9e014e02afe5de39d15e5389 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/2f944dee9e014e02afe5de39d15e5389 2023-04-08 02:54:23,990 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/cdecaa039d904ae79ef831ba5ac06485 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/cdecaa039d904ae79ef831ba5ac06485 2023-04-08 02:54:23,991 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d38b4e710235c58cc9479560817e4c8d/A of d38b4e710235c58cc9479560817e4c8d into 2f944dee9e014e02afe5de39d15e5389(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:23,991 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:23,991 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d., storeName=d38b4e710235c58cc9479560817e4c8d/A, priority=12, startTime=1680922463533; duration=0sec 2023-04-08 02:54:23,991 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:23,994 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d38b4e710235c58cc9479560817e4c8d/C of d38b4e710235c58cc9479560817e4c8d into cdecaa039d904ae79ef831ba5ac06485(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:23,995 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:23,995 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d., storeName=d38b4e710235c58cc9479560817e4c8d/C, priority=12, startTime=1680922463533; duration=0sec 2023-04-08 02:54:23,995 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:24,056 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/668ef7a7188e4a50b4f770dc4c1d5a07 2023-04-08 02:54:24,124 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:24,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57996 deadline: 1680922524124, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:24,125 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:24,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922524125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:24,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:24,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922524125, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:24,428 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:24,428 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:24,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922524428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:24,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922524428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:24,428 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:24,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57996 deadline: 1680922524428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:24,471 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/b13ccba2c64c47a88c8ed870de50d938 2023-04-08 02:54:24,476 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/e9f8c45ddb6449a38207a6d70e666332 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/e9f8c45ddb6449a38207a6d70e666332 2023-04-08 02:54:24,481 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/e9f8c45ddb6449a38207a6d70e666332, entries=150, sequenceid=165, filesize=25.3 K 2023-04-08 02:54:24,482 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/668ef7a7188e4a50b4f770dc4c1d5a07 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/668ef7a7188e4a50b4f770dc4c1d5a07 2023-04-08 02:54:24,497 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/668ef7a7188e4a50b4f770dc4c1d5a07, entries=150, sequenceid=165, filesize=11.8 K 2023-04-08 02:54:24,500 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/b13ccba2c64c47a88c8ed870de50d938 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/b13ccba2c64c47a88c8ed870de50d938 2023-04-08 02:54:24,510 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/b13ccba2c64c47a88c8ed870de50d938, entries=150, sequenceid=165, filesize=11.8 K 2023-04-08 02:54:24,511 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for d38b4e710235c58cc9479560817e4c8d in 918ms, sequenceid=165, compaction requested=false 2023-04-08 02:54:24,511 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:24,511 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:24,511 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:24,511 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:24,511 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:24,512 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:24,512 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:24,513 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:24,513 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:24,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:24,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:24,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:24,514 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:24,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:24,514 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:24,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:24,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:24,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:24,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:24,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:24,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11761c97[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:24,516 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:24,516 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:24,517 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:24,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:24,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:24,517 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:24,517 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:24,517 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:24,517 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:24,517 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:24,519 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:24,519 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:24,519 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:24,519 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:24,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:24,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:24,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:24,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:24,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:24,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:24,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:24,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:24,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:24,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:24,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:24,523 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:24,523 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:24,523 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:24,523 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:24,523 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:24,525 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:24,525 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:24,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:24,525 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:24,525 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:24,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:24,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:24,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:24,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:24,526 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:24,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:24,528 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:24,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:24,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:24,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:24,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:24,530 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:24,530 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:24,530 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:24,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:24,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:24,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:24,530 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:24,530 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:24,530 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:24,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:24,530 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:24,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:24,530 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:24,530 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:24,530 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:24,530 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:24,530 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:24,531 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:24,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:24,630 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:24,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:24,632 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:24,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:24,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:24,634 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:24,634 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:24,634 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:24,634 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:24,634 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:24,634 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:24,637 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:24,637 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:24,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:24,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:24,637 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:24,637 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:24,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:24,638 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:24,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:24,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:24,638 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:24,638 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:24,638 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:24,638 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:24,639 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:24,639 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:24,639 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:24,639 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:24,639 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:24,639 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:24,639 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d38b4e710235c58cc9479560817e4c8d 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-08 02:54:24,639 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=A 2023-04-08 02:54:24,639 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:24,639 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=B 2023-04-08 02:54:24,639 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:24,639 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=C 2023-04-08 02:54:24,639 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:24,656 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040858239c9a5fc641abb3570b8c54eac5e4 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040858239c9a5fc641abb3570b8c54eac5e4 2023-04-08 02:54:24,669 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=181, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/338061bf97124e4d98b960781cb9a383 2023-04-08 02:54:24,698 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/b0e1ac90b93a402b9a5cbe57b9d90cc7 2023-04-08 02:54:24,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:24,707 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:24,749 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:24,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922524748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:24,852 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:24,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922524850, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:24,932 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:24,933 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57996 deadline: 1680922524929, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:24,933 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:24,934 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922524932, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:24,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:24,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922524933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:25,055 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:25,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922525055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:25,114 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=181 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/6141f9392d684adea47326950dd4ec5d 2023-04-08 02:54:25,120 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/338061bf97124e4d98b960781cb9a383 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/338061bf97124e4d98b960781cb9a383 2023-04-08 02:54:25,125 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/338061bf97124e4d98b960781cb9a383, entries=150, sequenceid=181, filesize=25.3 K 2023-04-08 02:54:25,128 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/b0e1ac90b93a402b9a5cbe57b9d90cc7 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/b0e1ac90b93a402b9a5cbe57b9d90cc7 2023-04-08 02:54:25,133 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/b0e1ac90b93a402b9a5cbe57b9d90cc7, entries=150, sequenceid=181, filesize=11.8 K 2023-04-08 02:54:25,134 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/6141f9392d684adea47326950dd4ec5d as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/6141f9392d684adea47326950dd4ec5d 2023-04-08 02:54:25,147 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/6141f9392d684adea47326950dd4ec5d, entries=150, sequenceid=181, filesize=11.8 K 2023-04-08 02:54:25,148 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for d38b4e710235c58cc9479560817e4c8d in 509ms, sequenceid=181, compaction requested=true 2023-04-08 02:54:25,148 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:25,148 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:25,148 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:25,148 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:25,148 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:25,148 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:25,148 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:25,150 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:25,150 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:25,150 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:25,150 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:25,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:25,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:25,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:25,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6774c2c5[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:25,152 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:25,152 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:25,154 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:25,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:25,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:25,154 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:25,154 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:25,154 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:25,154 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:25,154 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:25,155 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:25,155 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:25,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:25,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:25,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:25,155 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:25,155 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:25,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:25,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:25,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:25,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:25,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:25,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:25,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:25,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:25,160 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:25,160 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:25,160 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:25,160 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:25,160 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:25,160 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:25,161 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:25,162 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:25,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:25,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:25,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:25,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:25,163 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:25,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:25,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:25,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:25,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:25,164 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:25,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:25,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:25,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:25,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:25,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:25,166 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:25,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:25,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:25,169 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:25,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:25,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:25,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:25,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:25,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:25,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:25,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:25,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:25,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:25,170 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:25,170 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:25,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:25,170 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:25,170 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:25,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:25,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:25,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:25,270 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:25,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:25,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:25,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:25,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:25,273 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:25,273 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:25,273 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:25,273 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:25,274 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:25,274 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:25,276 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:25,276 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:25,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:25,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:25,276 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:25,277 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:25,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:25,277 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:25,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:25,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:25,277 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:25,278 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:25,278 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:25,278 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:25,278 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:25,278 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:25,278 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:25,278 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:25,278 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:25,278 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:25,278 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d38b4e710235c58cc9479560817e4c8d 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-08 02:54:25,278 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=A 2023-04-08 02:54:25,279 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:25,279 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=B 2023-04-08 02:54:25,279 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:25,279 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=C 2023-04-08 02:54:25,279 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:25,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:25,359 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:25,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:25,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922525385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:25,490 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:25,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922525488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:25,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:25,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922525691, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:25,693 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408a5b7f57defa7432a915176ef0f3ae633 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408a5b7f57defa7432a915176ef0f3ae633 2023-04-08 02:54:25,819 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:25,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922525819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:25,822 DEBUG [Thread-994] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4135 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:54:25,937 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:25,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922525935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:25,939 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:25,939 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57996 deadline: 1680922525937, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:25,945 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:25,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922525943, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:25,995 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:25,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58036 deadline: 1680922525994, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:26,102 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=199, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/03da205c9283405592309a68adeb2667 2023-04-08 02:54:26,113 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/6136115768c44833a7295f54f914420d 2023-04-08 02:54:26,126 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=199 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/6853699a53bd4f1e92920fbb571aa1e2 2023-04-08 02:54:26,131 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/03da205c9283405592309a68adeb2667 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/03da205c9283405592309a68adeb2667 2023-04-08 02:54:26,135 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/03da205c9283405592309a68adeb2667, entries=150, sequenceid=199, filesize=25.3 K 2023-04-08 02:54:26,138 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/6136115768c44833a7295f54f914420d as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/6136115768c44833a7295f54f914420d 2023-04-08 02:54:26,142 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/6136115768c44833a7295f54f914420d, entries=150, sequenceid=199, filesize=11.8 K 2023-04-08 02:54:26,143 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/6853699a53bd4f1e92920fbb571aa1e2 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/6853699a53bd4f1e92920fbb571aa1e2 2023-04-08 02:54:26,147 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/6853699a53bd4f1e92920fbb571aa1e2, entries=150, sequenceid=199, filesize=11.8 K 2023-04-08 02:54:26,148 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for d38b4e710235c58cc9479560817e4c8d in 870ms, sequenceid=199, compaction requested=true 2023-04-08 02:54:26,148 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:26,148 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:26,148 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:26,148 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:26,148 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:26,148 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:26,148 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:26,150 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:26,150 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:26,150 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:26,150 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:26,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:26,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:26,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:26,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:26,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:26,153 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33d4a58d[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:26,153 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:26,153 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:26,154 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:26,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:26,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:26,154 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:26,154 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:26,154 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:26,155 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:26,155 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:26,157 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:26,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:26,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:26,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:26,157 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:26,157 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:26,157 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:26,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:26,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:26,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:26,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:26,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:26,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:26,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:26,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:26,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:26,159 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:26,160 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:26,160 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:26,160 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:26,160 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:26,161 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:26,161 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:26,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:26,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:26,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:26,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:26,161 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:26,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:26,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:26,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:26,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:26,162 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:26,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:26,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:26,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:26,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:26,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:26,164 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:26,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:26,164 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:26,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:26,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:26,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:26,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:26,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:26,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:26,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:26,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:26,169 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:26,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:26,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:26,169 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:26,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:26,169 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:26,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:26,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:26,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:26,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:26,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:26,269 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:26,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:26,272 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:26,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:26,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:26,273 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:26,273 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:26,273 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:26,273 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:26,274 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:26,274 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:26,277 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:26,277 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:26,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:26,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:26,277 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:26,277 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:26,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:26,278 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:26,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:26,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:26,278 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:26,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:26,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:26,280 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:26,280 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:26,280 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:26,280 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:26,280 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:26,280 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:26,281 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:26,281 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d38b4e710235c58cc9479560817e4c8d 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-08 02:54:26,281 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=A 2023-04-08 02:54:26,281 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:26,281 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=B 2023-04-08 02:54:26,281 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:26,281 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=C 2023-04-08 02:54:26,281 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:26,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:26,497 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:26,545 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:26,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922526541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:26,647 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:26,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922526646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:26,697 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408df0248050459447d8d4c2d47a45e7fce to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408df0248050459447d8d4c2d47a45e7fce 2023-04-08 02:54:26,853 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:26,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922526849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:27,104 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=213, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/ebf35745d2c44542be96b3cb82776b66 2023-04-08 02:54:27,114 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/38475076b31743548fb0adb564fde52f 2023-04-08 02:54:27,137 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=213 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/031d7681eb334ee698f82c114ec9a647 2023-04-08 02:54:27,142 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/ebf35745d2c44542be96b3cb82776b66 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/ebf35745d2c44542be96b3cb82776b66 2023-04-08 02:54:27,147 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/ebf35745d2c44542be96b3cb82776b66, entries=150, sequenceid=213, filesize=25.3 K 2023-04-08 02:54:27,148 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/38475076b31743548fb0adb564fde52f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/38475076b31743548fb0adb564fde52f 2023-04-08 02:54:27,152 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/38475076b31743548fb0adb564fde52f, entries=150, sequenceid=213, filesize=11.8 K 2023-04-08 02:54:27,157 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/031d7681eb334ee698f82c114ec9a647 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/031d7681eb334ee698f82c114ec9a647 2023-04-08 02:54:27,160 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:27,160 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922527156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:27,161 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/031d7681eb334ee698f82c114ec9a647, entries=150, sequenceid=213, filesize=11.8 K 2023-04-08 02:54:27,162 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for d38b4e710235c58cc9479560817e4c8d in 881ms, sequenceid=213, compaction requested=true 2023-04-08 02:54:27,163 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:27,163 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:27,163 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:27,163 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:27,163 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:27,166 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:27,166 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:27,168 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:27,168 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:27,168 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:27,168 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:27,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:27,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:27,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:27,170 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10cbbaef[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:27,170 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:27,170 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:27,171 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:27,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:27,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:27,172 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:27,172 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:27,172 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:27,172 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:27,172 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:27,173 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:27,173 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:27,173 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:27,173 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:27,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:27,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:27,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:27,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:27,175 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:27,175 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:27,175 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:27,175 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:27,175 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:27,177 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:27,177 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:27,177 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:27,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:27,177 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:27,177 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:27,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:27,178 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:27,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:27,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:27,180 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:27,180 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:27,180 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:27,180 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:27,181 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:27,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:27,181 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:27,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:27,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:27,180 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:27,181 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:27,181 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:27,181 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:27,181 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:27,181 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:27,181 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:27,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:27,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:27,184 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:27,281 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:27,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:27,284 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:27,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:27,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:27,286 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:27,286 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:27,286 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:27,286 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:27,286 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:27,286 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:27,289 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:27,289 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:27,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:27,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:27,289 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:27,289 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:27,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:27,290 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:27,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:27,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:27,290 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:27,296 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:27,296 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:27,296 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:27,296 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:27,296 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:27,308 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:27,308 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:27,308 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:27,308 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:27,308 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d38b4e710235c58cc9479560817e4c8d 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-08 02:54:27,308 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=A 2023-04-08 02:54:27,309 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:27,309 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=B 2023-04-08 02:54:27,309 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:27,309 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=C 2023-04-08 02:54:27,309 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:27,664 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:27,664 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:27,703 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:27,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922527699, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:27,734 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408263e64a5020c432ea56f5ff810552370 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408263e64a5020c432ea56f5ff810552370 2023-04-08 02:54:27,739 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=231, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/e24684b511a04dfe997c5e72a6cf51a8 2023-04-08 02:54:27,808 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:27,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922527805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:27,943 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:27,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922527941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:27,944 DEBUG [Thread-996] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4128 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:54:27,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:27,949 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922527948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:27,950 DEBUG [Thread-992] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4133 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:54:27,951 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:27,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57996 deadline: 1680922527949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:27,952 DEBUG [Thread-1000] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4134 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:54:28,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:28,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 153 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922528010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:28,149 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/68ca387b786446a1978de536f129462d 2023-04-08 02:54:28,313 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:28,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922528312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:28,560 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=231 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/a7daa1f6c6df4312980000d8e769dbd7 2023-04-08 02:54:28,566 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/e24684b511a04dfe997c5e72a6cf51a8 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/e24684b511a04dfe997c5e72a6cf51a8 2023-04-08 02:54:28,572 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/e24684b511a04dfe997c5e72a6cf51a8, entries=150, sequenceid=231, filesize=25.3 K 2023-04-08 02:54:28,574 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/68ca387b786446a1978de536f129462d as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/68ca387b786446a1978de536f129462d 2023-04-08 02:54:28,578 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/68ca387b786446a1978de536f129462d, entries=150, sequenceid=231, filesize=11.8 K 2023-04-08 02:54:28,579 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/a7daa1f6c6df4312980000d8e769dbd7 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a7daa1f6c6df4312980000d8e769dbd7 2023-04-08 02:54:28,588 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a7daa1f6c6df4312980000d8e769dbd7, entries=150, sequenceid=231, filesize=11.8 K 2023-04-08 02:54:28,590 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for d38b4e710235c58cc9479560817e4c8d in 1282ms, sequenceid=231, compaction requested=true 2023-04-08 02:54:28,590 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:28,590 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:28,590 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:28,590 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:28,590 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:28,596 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:28,596 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:28,598 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:28,598 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:28,598 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:28,598 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:28,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:28,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:28,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:28,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:28,600 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31788ec5[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:28,600 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:28,600 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:28,601 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:28,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:28,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:28,601 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:28,601 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:28,601 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:28,603 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:28,603 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:28,603 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:28,604 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:28,604 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:28,604 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:28,604 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:28,604 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:28,604 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:28,604 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:28,604 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:28,606 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:28,606 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:28,606 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:28,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:28,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:28,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:28,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:28,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:28,606 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:28,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:28,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:28,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:28,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:28,608 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:28,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:28,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:28,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:28,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:28,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:28,614 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:28,614 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:28,614 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:28,614 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:28,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:28,614 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:28,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:28,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:28,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:28,614 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:28,614 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:28,614 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:28,614 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:28,614 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:28,614 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:28,614 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:28,614 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:28,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:28,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:28,714 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:28,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:28,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:28,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:28,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:28,718 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:28,718 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:28,718 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:28,718 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:28,718 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:28,718 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:28,720 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:28,720 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:28,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:28,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:28,721 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:28,721 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:28,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:28,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:28,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:28,722 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:28,722 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:28,722 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:28,722 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:28,722 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:28,722 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:28,723 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:28,723 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:28,723 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:28,723 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:28,723 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d38b4e710235c58cc9479560817e4c8d 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-08 02:54:28,723 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=A 2023-04-08 02:54:28,723 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:28,723 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=B 2023-04-08 02:54:28,723 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:28,723 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=C 2023-04-08 02:54:28,723 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:28,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:28,818 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:28,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:28,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922528854, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:28,961 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:28,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922528960, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:29,155 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408920a7ecc42da4c85adcca4cb06584182 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408920a7ecc42da4c85adcca4cb06584182 2023-04-08 02:54:29,163 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:29,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922529163, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:29,182 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=245, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/22da8d5705404c668c64df3da840d24f 2023-04-08 02:54:29,195 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/45c20656a8f54cdaa3e1338e0a0dc529 2023-04-08 02:54:29,466 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:29,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922529464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:29,624 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=245 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/5598d76e6927482d923eb4b12d909e5b 2023-04-08 02:54:29,628 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/22da8d5705404c668c64df3da840d24f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/22da8d5705404c668c64df3da840d24f 2023-04-08 02:54:29,632 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/22da8d5705404c668c64df3da840d24f, entries=150, sequenceid=245, filesize=25.3 K 2023-04-08 02:54:29,633 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/45c20656a8f54cdaa3e1338e0a0dc529 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/45c20656a8f54cdaa3e1338e0a0dc529 2023-04-08 02:54:29,637 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/45c20656a8f54cdaa3e1338e0a0dc529, entries=150, sequenceid=245, filesize=11.8 K 2023-04-08 02:54:29,637 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/5598d76e6927482d923eb4b12d909e5b as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/5598d76e6927482d923eb4b12d909e5b 2023-04-08 02:54:29,641 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/5598d76e6927482d923eb4b12d909e5b, entries=150, sequenceid=245, filesize=11.8 K 2023-04-08 02:54:29,642 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for d38b4e710235c58cc9479560817e4c8d in 919ms, sequenceid=245, compaction requested=true 2023-04-08 02:54:29,642 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:29,642 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:29,642 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:29,642 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:29,642 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:29,642 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:29,642 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:29,643 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:29,643 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:29,644 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:29,644 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:29,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:29,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@705b00c5[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:29,645 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:29,645 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:29,646 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:29,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:29,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:29,646 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:29,646 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:29,646 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:29,646 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:29,646 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:29,648 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:29,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:29,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:29,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:29,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:29,650 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:29,650 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:29,650 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:29,650 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:29,650 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:29,650 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:29,653 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:29,653 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:29,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:29,653 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:29,653 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:29,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:29,655 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:29,660 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:29,660 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:29,660 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:29,660 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:29,660 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:29,660 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:29,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:29,660 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:29,660 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:29,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:29,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:29,660 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:29,660 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:29,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:29,660 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:29,660 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:29,761 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:29,761 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:29,763 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:29,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:29,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:29,765 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:29,765 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:29,765 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:29,765 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:29,765 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:29,765 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:29,768 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:29,768 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:29,768 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:29,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:29,768 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:29,768 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:29,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:29,769 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:29,769 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:29,769 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:29,769 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:29,769 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:29,769 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:29,770 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:29,770 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:29,770 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:29,770 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:29,770 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:29,770 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d38b4e710235c58cc9479560817e4c8d 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-08 02:54:29,770 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=A 2023-04-08 02:54:29,770 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:29,770 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=B 2023-04-08 02:54:29,771 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:29,771 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=C 2023-04-08 02:54:29,771 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:29,796 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040850821e081e844073ae40664a646fa8c4 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040850821e081e844073ae40664a646fa8c4 2023-04-08 02:54:29,808 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=263, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/ba40e967b6fa4f12b7c8b4c4ce277a0f 2023-04-08 02:54:29,826 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:29,826 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:29,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:29,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922529863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:29,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:29,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922529968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:29,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:29,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922529968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:30,175 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:30,176 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922530174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:30,226 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/173b9070f3194e42916eb91d7ef8dbec 2023-04-08 02:54:30,478 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:30,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922530477, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:30,637 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=263 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/a862ed86fed941189c3127c6ae737109 2023-04-08 02:54:30,642 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/ba40e967b6fa4f12b7c8b4c4ce277a0f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/ba40e967b6fa4f12b7c8b4c4ce277a0f 2023-04-08 02:54:30,648 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/ba40e967b6fa4f12b7c8b4c4ce277a0f, entries=150, sequenceid=263, filesize=25.4 K 2023-04-08 02:54:30,649 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/173b9070f3194e42916eb91d7ef8dbec as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/173b9070f3194e42916eb91d7ef8dbec 2023-04-08 02:54:30,658 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/173b9070f3194e42916eb91d7ef8dbec, entries=150, sequenceid=263, filesize=11.9 K 2023-04-08 02:54:30,659 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/a862ed86fed941189c3127c6ae737109 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a862ed86fed941189c3127c6ae737109 2023-04-08 02:54:30,663 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a862ed86fed941189c3127c6ae737109, entries=150, sequenceid=263, filesize=11.9 K 2023-04-08 02:54:30,664 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for d38b4e710235c58cc9479560817e4c8d in 894ms, sequenceid=263, compaction requested=true 2023-04-08 02:54:30,664 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:30,664 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:30,664 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:30,664 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:30,664 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:30,664 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:30,664 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:30,665 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:30,665 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:30,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:30,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:30,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:30,666 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:30,666 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:30,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:30,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:30,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:30,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:30,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:30,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:30,667 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@66353b8e[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:30,667 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:30,667 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:30,668 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:30,668 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:30,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:30,668 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:30,669 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:30,669 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:30,669 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:30,669 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:30,671 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:30,671 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:30,671 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:30,671 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:30,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:30,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:30,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:30,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:30,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:30,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:30,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:30,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:30,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:30,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:30,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:30,674 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:30,674 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:30,674 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:30,674 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:30,674 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:30,674 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:30,675 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:30,675 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:30,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:30,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:30,675 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:30,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:30,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:30,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:30,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:30,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:30,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:30,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:30,676 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:30,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:30,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:30,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:30,677 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:30,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:30,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:30,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:30,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:30,679 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:30,679 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:30,679 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:30,679 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:30,679 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:30,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:30,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:30,679 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:30,679 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:30,679 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:30,679 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:30,679 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:30,679 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:30,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:30,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:30,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:30,680 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:30,680 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:30,680 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:30,780 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:30,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:30,782 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:30,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:30,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:30,784 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:30,784 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:30,784 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:30,784 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:30,784 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:30,784 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:30,785 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:30,785 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:30,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:30,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:30,786 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:30,786 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:30,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:30,786 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:30,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:30,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:30,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:30,787 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:30,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:30,787 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:30,787 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:30,787 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:30,788 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:30,788 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:30,788 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:30,788 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:30,788 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d38b4e710235c58cc9479560817e4c8d 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-08 02:54:30,788 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=A 2023-04-08 02:54:30,788 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:30,788 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=B 2023-04-08 02:54:30,788 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:30,788 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=C 2023-04-08 02:54:30,788 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:30,814 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408cfa027c3f819483e9fcf472a40f28668 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408cfa027c3f819483e9fcf472a40f28668 2023-04-08 02:54:30,978 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:30,978 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:31,010 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:31,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922531007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,011 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:31,011 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922531010, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,114 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:31,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922531111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,116 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:31,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922531112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,224 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=277, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/4c88f2a65c5447339fc931212a3a36e5 2023-04-08 02:54:31,241 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/fd5cea4cdb314be98a07f92f6178f792 2023-04-08 02:54:31,257 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/efb9ca50a2da407aaf37464021164aba 2023-04-08 02:54:31,272 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/4c88f2a65c5447339fc931212a3a36e5 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/4c88f2a65c5447339fc931212a3a36e5 2023-04-08 02:54:31,276 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/4c88f2a65c5447339fc931212a3a36e5, entries=150, sequenceid=277, filesize=25.4 K 2023-04-08 02:54:31,279 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/fd5cea4cdb314be98a07f92f6178f792 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/fd5cea4cdb314be98a07f92f6178f792 2023-04-08 02:54:31,286 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/fd5cea4cdb314be98a07f92f6178f792, entries=150, sequenceid=277, filesize=11.9 K 2023-04-08 02:54:31,287 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/efb9ca50a2da407aaf37464021164aba as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/efb9ca50a2da407aaf37464021164aba 2023-04-08 02:54:31,294 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/efb9ca50a2da407aaf37464021164aba, entries=150, sequenceid=277, filesize=11.9 K 2023-04-08 02:54:31,294 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for d38b4e710235c58cc9479560817e4c8d in 506ms, sequenceid=277, compaction requested=true 2023-04-08 02:54:31,295 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:31,295 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:31,295 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:31,295 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:31,295 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,295 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:31,295 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:31,297 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,298 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:31,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:31,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:31,299 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,299 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:31,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:31,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:31,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:31,302 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2b64f965[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:31,302 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:31,302 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,303 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,304 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:31,304 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:31,304 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:31,304 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,304 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:31,305 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,305 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:31,305 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:31,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:31,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:31,305 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:31,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:31,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:31,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:31,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:31,310 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:31,310 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:31,310 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:31,310 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:31,310 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:31,310 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:31,312 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,312 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:31,312 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:31,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:31,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:31,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:31,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:31,314 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:31,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,315 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:31,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,319 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:31,319 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:31,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:31,319 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,319 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:31,319 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:31,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:31,320 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:31,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:31,320 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:31,320 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:31,320 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:31,320 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,321 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,321 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,321 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:31,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:31,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:31,323 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d38b4e710235c58cc9479560817e4c8d 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-08 02:54:31,324 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=A 2023-04-08 02:54:31,324 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:31,324 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=B 2023-04-08 02:54:31,324 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:31,324 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=C 2023-04-08 02:54:31,324 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:31,366 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304085a0f8334798d4a188421e989c11fa9e2 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304085a0f8334798d4a188421e989c11fa9e2 2023-04-08 02:54:31,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:31,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922531373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,386 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:31,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 200 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922531375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,420 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:31,421 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:31,424 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:31,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:31,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:31,434 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:31,434 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:31,434 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:31,434 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:31,435 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,435 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:31,436 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:31,436 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:31,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:31,436 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,436 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:31,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:31,437 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:31,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:31,438 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:31,439 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:31,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:31,440 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:31,440 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:31,440 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:31,440 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:31,440 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:31,440 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:31,440 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:31,441 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:31,441 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:31,441 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:31,441 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:31,441 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,441 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:31,441 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:31,442 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,442 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:31,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:31,443 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,443 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:31,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:31,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:31,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:31,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:31,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e9f446a[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:31,446 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:31,446 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,448 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,448 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,448 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:31,448 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:31,448 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:31,448 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:31,450 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,451 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:31,451 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:31,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:31,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:31,451 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:31,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:31,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:31,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:31,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:31,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:31,455 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:31,455 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:31,455 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:31,455 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:31,455 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:31,457 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,457 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:31,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:31,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:31,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,458 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:31,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:31,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:31,458 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:31,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,472 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:31,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,475 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:31,475 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:31,475 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:31,475 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:31,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:31,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:31,475 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:31,475 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:31,475 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,475 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,475 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,475 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:31,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:31,475 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:31,475 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:31,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:31,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:31,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922531487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,491 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:31,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922531487, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,576 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:31,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:31,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:31,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:31,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:31,579 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:31,579 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:31,579 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:31,579 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:31,579 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,579 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:31,580 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,581 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:31,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:31,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:31,581 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,581 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:31,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:31,581 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:31,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:31,582 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:31,582 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:31,582 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:31,582 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:31,582 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:31,582 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:31,582 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:31,582 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:31,582 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:31,582 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:31,582 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:31,582 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:31,583 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:31,583 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:31,583 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,583 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:31,583 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:31,587 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,587 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:31,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:31,588 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,588 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:31,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:31,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:31,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:31,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:31,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@766a3a54[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:31,589 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:31,589 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,590 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,590 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,590 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:31,590 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:31,590 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:31,590 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:31,591 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:31,591 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,592 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:31,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:31,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:31,592 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:31,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:31,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:31,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:31,594 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:31,594 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:31,594 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:31,594 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:31,594 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:31,594 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:31,595 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,595 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,595 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:31,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:31,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:31,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:31,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,600 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,601 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,605 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,605 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:31,605 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:31,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:31,605 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:31,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:31,605 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:31,605 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,605 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:31,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:31,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:31,605 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,605 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,605 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,605 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:31,605 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:31,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:31,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:31,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:31,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922531692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,695 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:31,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922531693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,706 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:31,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:31,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:31,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:31,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:31,716 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:31,716 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:31,716 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:31,716 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:31,717 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,717 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:31,718 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:31,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:31,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:31,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:31,719 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,719 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,720 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,720 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:31,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:31,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:31,720 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:31,720 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:31,720 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:31,720 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:31,720 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:31,720 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:31,720 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:31,721 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:31,721 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:31,721 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:31,721 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:31,721 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:31,721 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:31,721 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:31,721 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,721 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:31,721 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:31,722 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,722 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:31,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:31,723 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,723 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:31,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:31,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:31,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:31,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:31,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39cc865c[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:31,725 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:31,725 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,726 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,726 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:31,726 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:31,726 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:31,726 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,726 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:31,728 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,728 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:31,728 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:31,728 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:31,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:31,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:31,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:31,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:31,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:31,730 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:31,731 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:31,731 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:31,731 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:31,731 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:31,732 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,732 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:31,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:31,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,732 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:31,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:31,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:31,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:31,733 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:31,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,734 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:31,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,736 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,736 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:31,736 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:31,736 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:31,736 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:31,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:31,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:31,736 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,736 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,736 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,736 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,736 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:31,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:31,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:31,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:31,737 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:31,737 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:31,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:31,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:31,776 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=297, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/d6f00df893964b55939916c8d647edd2 2023-04-08 02:54:31,837 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:31,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:31,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:31,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:31,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:31,841 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:31,842 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:31,842 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:31,842 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:31,842 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,842 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:31,844 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:31,844 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:31,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:31,844 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,844 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:31,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:31,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:31,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:31,845 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:31,848 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:31,848 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:31,848 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:31,848 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:31,848 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:31,848 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:31,848 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:31,848 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:31,848 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:31,848 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:31,849 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:31,849 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:31,849 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:31,849 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,849 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:31,849 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:31,854 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,854 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:31,854 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,854 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:31,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:31,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:31,856 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35efefd1[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:31,856 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:31,856 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,857 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,857 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:31,857 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:31,857 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:31,857 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,857 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:31,858 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:31,858 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:31,858 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,858 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:31,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:31,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:31,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:31,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:31,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:31,861 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:31,861 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:31,861 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:31,861 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:31,861 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:31,861 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:31,862 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,862 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:31,862 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:31,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:31,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:31,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:31,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:31,863 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:31,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,865 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:31,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,870 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:31,870 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:31,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:31,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:31,870 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:31,870 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:31,870 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:31,870 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:31,870 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:31,870 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,871 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,871 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,870 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:31,871 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,871 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:31,871 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:31,871 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:31,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:31,966 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:31,967 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58068 deadline: 1680922531961, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,968 DEBUG [Thread-996] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:54:31,971 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:31,971 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:31,973 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:31,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:31,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:31,974 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:31,974 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:31,974 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:31,974 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:31,975 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,975 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:31,976 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,976 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:31,976 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:31,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:31,976 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,977 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:31,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:31,977 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:31,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:31,977 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:31,978 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:31,978 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:31,978 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:31,978 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:31,978 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:31,978 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:31,978 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:31,978 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:31,978 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:31,978 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:31,978 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:31,978 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:31,978 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:31,978 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,979 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:31,979 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:31,980 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,980 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:31,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:31,980 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,980 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:31,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:31,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:31,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,981 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:31,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,982 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58050 deadline: 1680922531981, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:31,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:31,982 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11f74112[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:31,982 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:31,982 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,984 DEBUG [Thread-992] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8167 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:54:31,986 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:31,987 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:31,987 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:31,987 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:31,988 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:31,988 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:31,988 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:31,989 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:31,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:57996 deadline: 1680922531988, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,991 DEBUG [Thread-1000] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8172 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:54:31,991 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,991 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:31,991 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:31,991 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:31,991 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:31,991 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:31,991 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:31,991 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:31,993 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,993 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,993 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:31,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:31,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:31,993 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:31,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:31,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:31,995 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:31,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:31,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:31,998 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:31,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922531996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:31,999 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:31,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:58052 deadline: 1680922531996, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:32,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,005 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:32,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:32,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,008 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:32,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:32,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:32,010 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,010 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:32,010 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:32,010 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,010 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:32,010 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:32,010 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:32,010 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:32,010 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:32,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:32,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:32,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:32,011 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:32,012 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:32,012 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:32,112 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:32,113 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:32,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:32,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:32,116 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:32,116 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:32,116 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:32,116 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:32,116 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:32,116 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:32,116 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:32,117 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:32,117 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:32,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:32,118 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,118 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:32,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:32,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:32,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:32,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:32,119 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:32,119 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:32,119 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:32,119 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:32,119 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:32,119 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:32,120 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:32,120 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:32,120 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:32,120 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:32,120 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:32,120 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:32,120 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:32,120 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:32,120 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,120 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:32,120 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:32,121 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:32,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:32,122 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:32,122 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:32,122 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:32,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:32,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:32,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:32,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:32,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:32,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36e871e2[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:32,123 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:32,123 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:32,124 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:32,124 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:32,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:32,124 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:32,124 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,124 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:32,124 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:32,125 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:32,127 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:32,127 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:32,127 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:32,128 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:32,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:32,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:32,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:32,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:32,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:32,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:32,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:32,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:32,130 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:32,130 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:32,130 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:32,130 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:32,130 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:32,132 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:32,132 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:32,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:32,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:32,132 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:32,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:32,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:32,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:32,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:32,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:32,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:32,133 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:32,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:32,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:32,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:32,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,135 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:32,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:32,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,137 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,137 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:32,137 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:32,137 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,137 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:32,137 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:32,137 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:32,137 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:32,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:32,137 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:32,138 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:32,138 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:32,138 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:32,138 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:32,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:32,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:32,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:32,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:32,204 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/a0c0d0013445410faf57e8697eb0d9f4 2023-04-08 02:54:32,220 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=297 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/c92700639dc649b19d57b17bd81b29ea 2023-04-08 02:54:32,230 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/d6f00df893964b55939916c8d647edd2 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/d6f00df893964b55939916c8d647edd2 2023-04-08 02:54:32,234 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/d6f00df893964b55939916c8d647edd2, entries=250, sequenceid=297, filesize=39.2 K 2023-04-08 02:54:32,235 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/a0c0d0013445410faf57e8697eb0d9f4 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/a0c0d0013445410faf57e8697eb0d9f4 2023-04-08 02:54:32,238 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:32,238 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:32,240 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/a0c0d0013445410faf57e8697eb0d9f4, entries=150, sequenceid=297, filesize=11.9 K 2023-04-08 02:54:32,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:32,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:32,241 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:32,241 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:32,241 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:32,242 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:32,242 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:32,244 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:32,244 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:32,246 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:32,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:32,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:32,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:32,246 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:32,247 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,247 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:32,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:32,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:32,247 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:32,248 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/c92700639dc649b19d57b17bd81b29ea as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/c92700639dc649b19d57b17bd81b29ea 2023-04-08 02:54:32,249 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:32,249 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:32,249 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:32,249 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:32,249 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:32,249 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:32,249 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:32,249 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:32,249 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:32,249 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:32,249 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:32,249 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:32,249 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:32,250 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,250 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:32,250 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:32,253 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,253 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:32,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:32,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:32,254 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:32,254 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:32,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:32,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:32,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bcbef05[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:32,255 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:32,255 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:32,257 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:32,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:32,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:32,257 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:32,257 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:32,257 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:32,257 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,257 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:32,259 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:32,259 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:32,259 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,259 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:32,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:32,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:32,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:32,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:32,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:32,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:32,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:32,262 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:32,262 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:32,262 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:32,262 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:32,262 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:32,264 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:32,264 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:32,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:32,264 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:32,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:32,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:32,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:32,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:32,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:32,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:32,265 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:32,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:32,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:32,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:32,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:32,266 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:32,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:32,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,268 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/c92700639dc649b19d57b17bd81b29ea, entries=150, sequenceid=297, filesize=11.9 K 2023-04-08 02:54:32,269 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for d38b4e710235c58cc9479560817e4c8d in 946ms, sequenceid=297, compaction requested=true 2023-04-08 02:54:32,269 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:32,269 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:32,269 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-04-08 02:54:32,269 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-04-08 02:54:32,270 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,270 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:32,270 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:32,270 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,270 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:32,270 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:32,270 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:32,270 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:32,269 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:32,270 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:32,270 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:32,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:32,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:32,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:32,270 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:32,270 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:32,270 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:32,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:32,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:32,271 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:32,271 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:32,274 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 273768 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-04-08 02:54:32,274 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 121084 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-04-08 02:54:32,274 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): d38b4e710235c58cc9479560817e4c8d/A is initiating minor compaction (all files) 2023-04-08 02:54:32,274 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d38b4e710235c58cc9479560817e4c8d/B is initiating minor compaction (all files) 2023-04-08 02:54:32,274 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d38b4e710235c58cc9479560817e4c8d/A in TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:32,274 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d38b4e710235c58cc9479560817e4c8d/B in TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:32,274 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/2f944dee9e014e02afe5de39d15e5389, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/e9f8c45ddb6449a38207a6d70e666332, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/338061bf97124e4d98b960781cb9a383, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/03da205c9283405592309a68adeb2667, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/ebf35745d2c44542be96b3cb82776b66, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/e24684b511a04dfe997c5e72a6cf51a8, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/22da8d5705404c668c64df3da840d24f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/ba40e967b6fa4f12b7c8b4c4ce277a0f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/4c88f2a65c5447339fc931212a3a36e5, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/d6f00df893964b55939916c8d647edd2] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp, totalSize=267.4 K 2023-04-08 02:54:32,274 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/66855a5eb08041cebf8d309d1bace3ef, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/668ef7a7188e4a50b4f770dc4c1d5a07, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/b0e1ac90b93a402b9a5cbe57b9d90cc7, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/6136115768c44833a7295f54f914420d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/38475076b31743548fb0adb564fde52f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/68ca387b786446a1978de536f129462d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/45c20656a8f54cdaa3e1338e0a0dc529, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/173b9070f3194e42916eb91d7ef8dbec, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/fd5cea4cdb314be98a07f92f6178f792, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/a0c0d0013445410faf57e8697eb0d9f4] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp, totalSize=118.2 K 2023-04-08 02:54:32,275 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 2f944dee9e014e02afe5de39d15e5389, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1680922461685 2023-04-08 02:54:32,275 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 66855a5eb08041cebf8d309d1bace3ef, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1680922461685 2023-04-08 02:54:32,275 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting e9f8c45ddb6449a38207a6d70e666332, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680922462661 2023-04-08 02:54:32,276 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 668ef7a7188e4a50b4f770dc4c1d5a07, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680922462661 2023-04-08 02:54:32,276 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 338061bf97124e4d98b960781cb9a383, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680922463808 2023-04-08 02:54:32,276 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting b0e1ac90b93a402b9a5cbe57b9d90cc7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680922463808 2023-04-08 02:54:32,276 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 03da205c9283405592309a68adeb2667, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1680922464715 2023-04-08 02:54:32,276 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 6136115768c44833a7295f54f914420d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1680922464715 2023-04-08 02:54:32,277 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting ebf35745d2c44542be96b3cb82776b66, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680922465371 2023-04-08 02:54:32,277 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 38475076b31743548fb0adb564fde52f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680922465371 2023-04-08 02:54:32,277 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting e24684b511a04dfe997c5e72a6cf51a8, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1680922466530 2023-04-08 02:54:32,277 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 68ca387b786446a1978de536f129462d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1680922466530 2023-04-08 02:54:32,278 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 22da8d5705404c668c64df3da840d24f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1680922467691 2023-04-08 02:54:32,278 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting ba40e967b6fa4f12b7c8b4c4ce277a0f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1680922468839 2023-04-08 02:54:32,278 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 45c20656a8f54cdaa3e1338e0a0dc529, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1680922467691 2023-04-08 02:54:32,279 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 4c88f2a65c5447339fc931212a3a36e5, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1680922469843 2023-04-08 02:54:32,279 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 173b9070f3194e42916eb91d7ef8dbec, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1680922468839 2023-04-08 02:54:32,279 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting d6f00df893964b55939916c8d647edd2, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1680922471005 2023-04-08 02:54:32,280 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting fd5cea4cdb314be98a07f92f6178f792, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1680922469843 2023-04-08 02:54:32,280 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting a0c0d0013445410faf57e8697eb0d9f4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1680922471005 2023-04-08 02:54:32,319 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d38b4e710235c58cc9479560817e4c8d#B#compaction#189 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:32,328 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): d38b4e710235c58cc9479560817e4c8d#A#compaction#188 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:32,361 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/772cdd6afb064d0cb4bd1270a36a8baf as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/772cdd6afb064d0cb4bd1270a36a8baf 2023-04-08 02:54:32,371 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:32,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:32,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:32,378 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in d38b4e710235c58cc9479560817e4c8d/B of d38b4e710235c58cc9479560817e4c8d into 772cdd6afb064d0cb4bd1270a36a8baf(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:32,378 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:32,378 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d., storeName=d38b4e710235c58cc9479560817e4c8d/B, priority=6, startTime=1680922472269; duration=0sec 2023-04-08 02:54:32,378 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:32,378 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 10 store files, 0 compacting, 10 eligible, 16 blocking 2023-04-08 02:54:32,383 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 10 files of size 121084 starting at candidate #0 after considering 36 permutations with 36 in ratio 2023-04-08 02:54:32,383 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d38b4e710235c58cc9479560817e4c8d/C is initiating minor compaction (all files) 2023-04-08 02:54:32,383 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d38b4e710235c58cc9479560817e4c8d/C in TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:32,383 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/cdecaa039d904ae79ef831ba5ac06485, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/b13ccba2c64c47a88c8ed870de50d938, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/6141f9392d684adea47326950dd4ec5d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/6853699a53bd4f1e92920fbb571aa1e2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/031d7681eb334ee698f82c114ec9a647, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a7daa1f6c6df4312980000d8e769dbd7, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/5598d76e6927482d923eb4b12d909e5b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a862ed86fed941189c3127c6ae737109, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/efb9ca50a2da407aaf37464021164aba, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/c92700639dc649b19d57b17bd81b29ea] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp, totalSize=118.2 K 2023-04-08 02:54:32,383 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting cdecaa039d904ae79ef831ba5ac06485, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=146, earliestPutTs=1680922461685 2023-04-08 02:54:32,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:32,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:32,384 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting b13ccba2c64c47a88c8ed870de50d938, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680922462661 2023-04-08 02:54:32,384 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:32,384 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:32,384 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:32,384 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:32,385 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 6141f9392d684adea47326950dd4ec5d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=181, earliestPutTs=1680922463808 2023-04-08 02:54:32,385 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:32,385 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:32,385 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 6853699a53bd4f1e92920fbb571aa1e2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=199, earliestPutTs=1680922464715 2023-04-08 02:54:32,385 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 031d7681eb334ee698f82c114ec9a647, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=213, earliestPutTs=1680922465371 2023-04-08 02:54:32,386 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:32,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:32,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:32,386 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting a7daa1f6c6df4312980000d8e769dbd7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=231, earliestPutTs=1680922466530 2023-04-08 02:54:32,386 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:32,387 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:32,387 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 5598d76e6927482d923eb4b12d909e5b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=245, earliestPutTs=1680922467691 2023-04-08 02:54:32,387 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,387 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:32,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:32,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:32,387 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting a862ed86fed941189c3127c6ae737109, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=263, earliestPutTs=1680922468839 2023-04-08 02:54:32,387 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:32,388 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting efb9ca50a2da407aaf37464021164aba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1680922469843 2023-04-08 02:54:32,389 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting c92700639dc649b19d57b17bd81b29ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1680922471005 2023-04-08 02:54:32,397 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:32,397 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:32,397 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:32,397 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:32,397 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:32,400 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:32,400 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:32,400 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:32,400 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:32,400 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d38b4e710235c58cc9479560817e4c8d 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-08 02:54:32,400 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=A 2023-04-08 02:54:32,401 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:32,401 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=B 2023-04-08 02:54:32,401 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:32,401 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=C 2023-04-08 02:54:32,401 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:32,420 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d38b4e710235c58cc9479560817e4c8d#C#compaction#190 average throughput is 1.64 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:32,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:32,504 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:32,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922472572, max timestamp=1680922472583Num uniques 859; 2023-04-08 02:54:32,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922472572, max timestamp=1680922472583Num uniques 859; 2023-04-08 02:54:32,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:54:32,589 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:54:32,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922472572, max timestamp=1680922472583Num uniques 859; 2023-04-08 02:54:32,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:54:32,644 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:32,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58052 deadline: 1680922532639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,645 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:32,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 225 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922532639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:32,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58052 deadline: 1680922532746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,748 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:32,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922532746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,835 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/d37cfa71dbae4976bf757589dca25950 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/d37cfa71dbae4976bf757589dca25950 2023-04-08 02:54:32,839 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in d38b4e710235c58cc9479560817e4c8d/A of d38b4e710235c58cc9479560817e4c8d into d37cfa71dbae4976bf757589dca25950(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:32,840 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:32,840 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d., storeName=d38b4e710235c58cc9479560817e4c8d/A, priority=6, startTime=1680922472269; duration=0sec 2023-04-08 02:54:32,840 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:32,851 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/7ed1462a85d147e99fa87ad4a3585454 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/7ed1462a85d147e99fa87ad4a3585454 2023-04-08 02:54:32,855 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 10 (all) file(s) in d38b4e710235c58cc9479560817e4c8d/C of d38b4e710235c58cc9479560817e4c8d into 7ed1462a85d147e99fa87ad4a3585454(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:32,855 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:32,855 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d., storeName=d38b4e710235c58cc9479560817e4c8d/C, priority=6, startTime=1680922472270; duration=0sec 2023-04-08 02:54:32,855 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:32,882 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408e1d8dd7e6d9942a4a7fc4fee111ff236 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408e1d8dd7e6d9942a4a7fc4fee111ff236 2023-04-08 02:54:32,891 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=310, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/ed3dd768c5304e98a7e22e0111fc3e29 2023-04-08 02:54:32,911 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/920418ad8e434abab51f0ab0a6991687 2023-04-08 02:54:32,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:32,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922532949, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:32,950 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:32,950 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58052 deadline: 1680922532950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,251 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:33,252 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922533251, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,255 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:33,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58052 deadline: 1680922533253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,322 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=310 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/daa0654f7fb7410e91e48c47eacc9c53 2023-04-08 02:54:33,331 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/ed3dd768c5304e98a7e22e0111fc3e29 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/ed3dd768c5304e98a7e22e0111fc3e29 2023-04-08 02:54:33,343 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/ed3dd768c5304e98a7e22e0111fc3e29, entries=100, sequenceid=310, filesize=18.6 K 2023-04-08 02:54:33,344 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/920418ad8e434abab51f0ab0a6991687 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/920418ad8e434abab51f0ab0a6991687 2023-04-08 02:54:33,356 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/920418ad8e434abab51f0ab0a6991687, entries=100, sequenceid=310, filesize=9.5 K 2023-04-08 02:54:33,357 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/daa0654f7fb7410e91e48c47eacc9c53 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/daa0654f7fb7410e91e48c47eacc9c53 2023-04-08 02:54:33,362 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/daa0654f7fb7410e91e48c47eacc9c53, entries=100, sequenceid=310, filesize=9.5 K 2023-04-08 02:54:33,363 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for d38b4e710235c58cc9479560817e4c8d in 963ms, sequenceid=310, compaction requested=false 2023-04-08 02:54:33,363 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:33,363 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:33,363 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:33,363 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:33,363 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,363 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:33,363 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:33,365 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:33,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:33,366 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:33,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:33,366 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:33,366 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:33,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:33,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:33,370 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@211004ca[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:33,370 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:33,370 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:33,372 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:33,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:33,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:33,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:33,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:33,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:33,372 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,372 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:33,376 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:33,376 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,376 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:33,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:33,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:33,376 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:33,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:33,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:33,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:33,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:33,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:33,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:33,379 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:33,379 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:33,379 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:33,379 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:33,379 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:33,379 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:33,380 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:33,380 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:33,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:33,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:33,381 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:33,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:33,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:33,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:33,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:33,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:33,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:33,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:33,382 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:33,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:33,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:33,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,386 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:33,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:33,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,388 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,388 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:33,388 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:33,388 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,388 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:33,388 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:33,388 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:33,388 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:33,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:33,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:33,388 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:33,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:33,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:33,388 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:33,388 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:33,388 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:33,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:33,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:33,489 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:33,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:33,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:33,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:33,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:33,492 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:33,492 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:33,492 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:33,492 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:33,492 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:33,492 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:33,495 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:33,495 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:33,495 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,495 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:33,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:33,496 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:33,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:33,496 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:33,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:33,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:33,497 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:33,497 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:33,497 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:33,497 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:33,497 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:33,497 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:33,497 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:33,497 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d38b4e710235c58cc9479560817e4c8d 3/3 column families, dataSize=120.76 KB heapSize=345.14 KB 2023-04-08 02:54:33,497 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=A 2023-04-08 02:54:33,497 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:54:33,497 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=B 2023-04-08 02:54:33,497 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:54:33,497 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=C 2023-04-08 02:54:33,497 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:54:33,521 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408d09f07d22330422c8ec752f4871d265b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408d09f07d22330422c8ec752f4871d265b 2023-04-08 02:54:33,527 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=347, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/69ea031980564edd8934cd77d6d2f990 2023-04-08 02:54:33,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:33,754 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:33,797 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:33,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 155 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58052 deadline: 1680922533794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,801 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:33,801 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922533797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,899 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:33,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58052 deadline: 1680922533898, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:33,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922533902, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,939 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/c1c8b982cb0a43dcbfb90dd484852dd6 2023-04-08 02:54:33,954 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/1b08a62ecda2431da50465118bb80ae3 2023-04-08 02:54:33,960 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/69ea031980564edd8934cd77d6d2f990 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/69ea031980564edd8934cd77d6d2f990 2023-04-08 02:54:33,971 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/69ea031980564edd8934cd77d6d2f990, entries=150, sequenceid=347, filesize=25.4 K 2023-04-08 02:54:33,972 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/c1c8b982cb0a43dcbfb90dd484852dd6 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/c1c8b982cb0a43dcbfb90dd484852dd6 2023-04-08 02:54:33,979 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/c1c8b982cb0a43dcbfb90dd484852dd6, entries=150, sequenceid=347, filesize=11.9 K 2023-04-08 02:54:33,980 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/1b08a62ecda2431da50465118bb80ae3 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/1b08a62ecda2431da50465118bb80ae3 2023-04-08 02:54:33,983 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/1b08a62ecda2431da50465118bb80ae3, entries=150, sequenceid=347, filesize=11.9 K 2023-04-08 02:54:33,984 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~345.09 KB/353376, currentSize=60.38 KB/61830 for d38b4e710235c58cc9479560817e4c8d in 487ms, sequenceid=347, compaction requested=true 2023-04-08 02:54:33,984 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:33,984 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:33,984 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:33,985 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:33,985 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,985 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:33,985 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:33,986 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:33,986 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:33,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:33,986 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:33,987 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:33,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:33,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:33,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:33,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:33,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:33,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ac43e2b[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:33,989 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:33,989 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:33,991 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,991 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:33,992 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:33,992 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:33,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:33,992 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:33,992 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:33,992 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:33,993 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:33,993 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:33,993 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,993 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:33,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:33,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:33,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:33,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:33,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:33,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:33,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:33,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:33,996 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:33,996 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:33,996 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:33,996 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:33,996 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:33,996 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:33,998 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:33,998 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:33,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:33,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:33,998 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:33,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:33,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:33,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:33,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:33,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:33,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:33,998 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:33,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:33,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:33,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:33,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:34,000 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:34,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:34,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,003 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:34,003 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:34,003 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:34,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:34,003 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,003 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:34,003 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:34,003 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,003 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:34,003 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:34,004 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:34,004 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:34,004 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:34,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:34,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:34,004 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:34,004 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:34,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:34,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:34,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:34,103 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d38b4e710235c58cc9479560817e4c8d 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-08 02:54:34,103 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=A 2023-04-08 02:54:34,103 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:34,103 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=B 2023-04-08 02:54:34,103 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:34,103 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=C 2023-04-08 02:54:34,103 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:34,104 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:34,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:34,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:34,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:34,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:34,108 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:34,108 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:34,108 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:34,108 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:34,108 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:34,108 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:34,110 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,110 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:34,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:34,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:34,111 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,111 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:34,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:34,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:34,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:34,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:34,112 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:34,116 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:34,116 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:34,116 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:34,116 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:34,116 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:34,116 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:34,116 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:34,116 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:34,116 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:34,117 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:34,117 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:34,117 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:34,117 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:34,117 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,117 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:34,117 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:34,118 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,119 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:34,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:34,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:34,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:34,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:34,119 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:34,119 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:34,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:34,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:34,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:34,123 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fd629f0[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:34,123 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:34,123 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:34,126 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:34,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:34,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:34,127 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:34,127 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:34,127 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:34,127 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,127 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:34,128 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,128 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:34,128 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:34,128 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:34,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:34,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:34,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:34,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:34,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:34,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:34,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:34,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:34,133 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:34,133 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:34,134 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:34,134 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:34,134 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:34,134 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:34,136 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:34,136 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:34,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:34,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:34,136 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:34,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:34,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:34,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:34,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:34,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:34,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:34,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:34,138 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:34,138 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:34,138 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58052 deadline: 1680922534134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:34,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:34,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922534134, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,139 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:34,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:34,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:34,142 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:34,142 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:34,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:34,142 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:34,142 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,142 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:34,142 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:34,142 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,142 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:34,142 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:34,142 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:34,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:34,143 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:34,143 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:34,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:34,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:34,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:34,144 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:34,144 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:34,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:34,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922534240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:34,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58052 deadline: 1680922534240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,244 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:34,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:34,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:34,247 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:34,248 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:34,248 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:34,248 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:34,248 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:34,248 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:34,248 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:34,248 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:34,249 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:34,249 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,249 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:34,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:34,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:34,250 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,250 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:34,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:34,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:34,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:34,251 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:34,251 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:34,251 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:34,251 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:34,251 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:34,251 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:34,251 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:34,251 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:34,251 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:34,251 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. started... 2023-04-08 02:54:34,251 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. as already flushing 2023-04-08 02:54:34,252 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:34,252 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:34,252 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:34,252 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,252 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:34,252 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:34,255 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,255 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:34,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:34,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:34,255 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:34,255 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:34,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:34,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:34,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:34,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:34,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:34,257 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4bd142c8[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:34,257 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:34,257 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:34,258 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:34,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:34,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:34,258 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:34,258 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:34,258 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:34,259 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,259 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:34,260 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:34,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:34,260 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:34,260 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:34,260 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:34,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:34,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:34,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:34,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:34,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:34,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:34,262 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:34,262 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:34,262 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:34,262 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:34,262 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:34,262 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:34,264 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:34,264 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:34,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:34,264 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:34,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:34,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:34,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:34,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:34,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:34,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:34,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:34,265 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:34,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:34,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:34,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:34,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,268 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:34,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:34,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,278 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:34,278 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:34,278 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:34,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:34,278 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:34,278 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:34,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:34,278 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:34,278 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:34,279 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,279 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:34,279 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:34,279 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:34,279 DEBUG [Thread-1002] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:34,279 DEBUG [Thread-1002] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:34,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:34,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:34,357 DEBUG [Thread-1007] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4a0fd7b4 to 127.0.0.1:64826 2023-04-08 02:54:34,357 DEBUG [Thread-1007] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:54:34,357 DEBUG [Thread-1003] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x498694cb to 127.0.0.1:64826 2023-04-08 02:54:34,358 DEBUG [Thread-1003] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:54:34,358 DEBUG [Thread-1005] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x421e5356 to 127.0.0.1:64826 2023-04-08 02:54:34,359 DEBUG [Thread-1005] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:54:34,362 DEBUG [Thread-1009] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a7b66a3 to 127.0.0.1:64826 2023-04-08 02:54:34,362 DEBUG [Thread-1009] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:54:34,379 DEBUG [Thread-1002] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:34,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:34,443 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:34,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 256 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922534443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,444 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:34,444 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58052 deadline: 1680922534443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,531 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408c64fb392d5f34646ae75fded3d5d0d64 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408c64fb392d5f34646ae75fded3d5d0d64 2023-04-08 02:54:34,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:34,747 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:34,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 258 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922534747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58052 deadline: 1680922534747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:34,936 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=360, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/e26f12580f664201ae043a244d66ba77 2023-04-08 02:54:35,249 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:35,249 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:35,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 260 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58036 deadline: 1680922535249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:35,250 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:58052 deadline: 1680922535249, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:35,346 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/0d8dcaceb83c48659be94d04ed62413a 2023-04-08 02:54:35,755 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=360 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/a045c4aa94744b8fbc5b1ae14e7495d1 2023-04-08 02:54:35,759 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/e26f12580f664201ae043a244d66ba77 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/e26f12580f664201ae043a244d66ba77 2023-04-08 02:54:35,764 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/e26f12580f664201ae043a244d66ba77, entries=200, sequenceid=360, filesize=32.3 K 2023-04-08 02:54:35,765 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/0d8dcaceb83c48659be94d04ed62413a as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/0d8dcaceb83c48659be94d04ed62413a 2023-04-08 02:54:35,768 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/0d8dcaceb83c48659be94d04ed62413a, entries=150, sequenceid=360, filesize=11.9 K 2023-04-08 02:54:35,768 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/a045c4aa94744b8fbc5b1ae14e7495d1 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a045c4aa94744b8fbc5b1ae14e7495d1 2023-04-08 02:54:35,772 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a045c4aa94744b8fbc5b1ae14e7495d1, entries=150, sequenceid=360, filesize=11.9 K 2023-04-08 02:54:35,773 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for d38b4e710235c58cc9479560817e4c8d in 1670ms, sequenceid=360, compaction requested=true 2023-04-08 02:54:35,773 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:35,773 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:35,773 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:35,773 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:54:35,773 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:35,773 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:54:35,774 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104790 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:54:35,774 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46914 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:54:35,774 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): d38b4e710235c58cc9479560817e4c8d/A is initiating minor compaction (all files) 2023-04-08 02:54:35,774 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d38b4e710235c58cc9479560817e4c8d/B is initiating minor compaction (all files) 2023-04-08 02:54:35,774 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d38b4e710235c58cc9479560817e4c8d/A in TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:35,774 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d38b4e710235c58cc9479560817e4c8d/B in TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:35,774 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/d37cfa71dbae4976bf757589dca25950, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/ed3dd768c5304e98a7e22e0111fc3e29, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/69ea031980564edd8934cd77d6d2f990, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/e26f12580f664201ae043a244d66ba77] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp, totalSize=102.3 K 2023-04-08 02:54:35,774 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/772cdd6afb064d0cb4bd1270a36a8baf, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/920418ad8e434abab51f0ab0a6991687, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/c1c8b982cb0a43dcbfb90dd484852dd6, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/0d8dcaceb83c48659be94d04ed62413a] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp, totalSize=45.8 K 2023-04-08 02:54:35,775 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting d37cfa71dbae4976bf757589dca25950, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1680922471005 2023-04-08 02:54:35,775 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 772cdd6afb064d0cb4bd1270a36a8baf, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1680922471005 2023-04-08 02:54:35,775 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting ed3dd768c5304e98a7e22e0111fc3e29, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680922471361 2023-04-08 02:54:35,775 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 920418ad8e434abab51f0ab0a6991687, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680922471361 2023-04-08 02:54:35,775 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 69ea031980564edd8934cd77d6d2f990, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1680922472623 2023-04-08 02:54:35,775 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting c1c8b982cb0a43dcbfb90dd484852dd6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1680922472623 2023-04-08 02:54:35,775 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting e26f12580f664201ae043a244d66ba77, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680922473764 2023-04-08 02:54:35,776 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 0d8dcaceb83c48659be94d04ed62413a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680922473764 2023-04-08 02:54:35,782 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d38b4e710235c58cc9479560817e4c8d#B#compaction#200 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:35,785 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): d38b4e710235c58cc9479560817e4c8d#A#compaction#201 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:36,193 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/d6be5bcff0074e3f8c087e01974c7c8f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/d6be5bcff0074e3f8c087e01974c7c8f 2023-04-08 02:54:36,197 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d38b4e710235c58cc9479560817e4c8d/B of d38b4e710235c58cc9479560817e4c8d into d6be5bcff0074e3f8c087e01974c7c8f(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:36,197 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:36,197 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d., storeName=d38b4e710235c58cc9479560817e4c8d/B, priority=12, startTime=1680922475773; duration=0sec 2023-04-08 02:54:36,198 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:36,198 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:54:36,199 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46914 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:54:36,199 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d38b4e710235c58cc9479560817e4c8d/C is initiating minor compaction (all files) 2023-04-08 02:54:36,199 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d38b4e710235c58cc9479560817e4c8d/C in TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:36,199 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/7ed1462a85d147e99fa87ad4a3585454, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/daa0654f7fb7410e91e48c47eacc9c53, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/1b08a62ecda2431da50465118bb80ae3, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a045c4aa94744b8fbc5b1ae14e7495d1] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp, totalSize=45.8 K 2023-04-08 02:54:36,199 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 7ed1462a85d147e99fa87ad4a3585454, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=297, earliestPutTs=1680922471005 2023-04-08 02:54:36,200 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting daa0654f7fb7410e91e48c47eacc9c53, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=310, earliestPutTs=1680922471361 2023-04-08 02:54:36,200 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 1b08a62ecda2431da50465118bb80ae3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1680922472623 2023-04-08 02:54:36,200 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/6a074ef96fb44dfab4433cf930735d97 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/6a074ef96fb44dfab4433cf930735d97 2023-04-08 02:54:36,200 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting a045c4aa94744b8fbc5b1ae14e7495d1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=360, earliestPutTs=1680922473764 2023-04-08 02:54:36,204 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d38b4e710235c58cc9479560817e4c8d/A of d38b4e710235c58cc9479560817e4c8d into 6a074ef96fb44dfab4433cf930735d97(size=26.2 K), total size for store is 26.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:36,204 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:36,204 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d., storeName=d38b4e710235c58cc9479560817e4c8d/A, priority=12, startTime=1680922475773; duration=0sec 2023-04-08 02:54:36,204 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:36,208 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d38b4e710235c58cc9479560817e4c8d#C#compaction#202 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:36,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:36,256 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d38b4e710235c58cc9479560817e4c8d 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-08 02:54:36,256 DEBUG [Thread-998] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x370947cb to 127.0.0.1:64826 2023-04-08 02:54:36,256 DEBUG [Thread-994] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x02bab6b2 to 127.0.0.1:64826 2023-04-08 02:54:36,256 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=A 2023-04-08 02:54:36,257 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:54:36,257 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=B 2023-04-08 02:54:36,257 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:54:36,257 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=C 2023-04-08 02:54:36,257 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:54:36,256 DEBUG [Thread-998] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:54:36,256 DEBUG [Thread-994] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:54:36,616 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/17eefddb98f04f2c8bda955733bdaa15 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/17eefddb98f04f2c8bda955733bdaa15 2023-04-08 02:54:36,620 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d38b4e710235c58cc9479560817e4c8d/C of d38b4e710235c58cc9479560817e4c8d into 17eefddb98f04f2c8bda955733bdaa15(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:36,620 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:36,620 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d., storeName=d38b4e710235c58cc9479560817e4c8d/C, priority=12, startTime=1680922475773; duration=0sec 2023-04-08 02:54:36,620 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:36,670 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040822746b48639d4e368190497c00f2a962 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040822746b48639d4e368190497c00f2a962 2023-04-08 02:54:37,075 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=383, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/8a033f311b804ef98fb4009771d0f9df 2023-04-08 02:54:37,485 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=383 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/18fa9bbad11c4d8199adad343ff44549 2023-04-08 02:54:37,895 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=383 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/414507a808ec4e35a046fb503d3dea0d 2023-04-08 02:54:37,899 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/8a033f311b804ef98fb4009771d0f9df as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/8a033f311b804ef98fb4009771d0f9df 2023-04-08 02:54:37,902 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/8a033f311b804ef98fb4009771d0f9df, entries=150, sequenceid=383, filesize=25.4 K 2023-04-08 02:54:37,903 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/18fa9bbad11c4d8199adad343ff44549 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/18fa9bbad11c4d8199adad343ff44549 2023-04-08 02:54:37,907 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/18fa9bbad11c4d8199adad343ff44549, entries=150, sequenceid=383, filesize=11.9 K 2023-04-08 02:54:37,907 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/414507a808ec4e35a046fb503d3dea0d as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/414507a808ec4e35a046fb503d3dea0d 2023-04-08 02:54:37,911 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/414507a808ec4e35a046fb503d3dea0d, entries=150, sequenceid=383, filesize=11.9 K 2023-04-08 02:54:37,912 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~360 KB/368640, currentSize=0 B/0 for d38b4e710235c58cc9479560817e4c8d in 1656ms, sequenceid=383, compaction requested=false 2023-04-08 02:54:37,912 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:39,728 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-08 02:54:41,971 DEBUG [Thread-996] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x24bd3872 to 127.0.0.1:64826 2023-04-08 02:54:41,971 DEBUG [Thread-996] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:54:42,068 DEBUG [Thread-992] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34135972 to 127.0.0.1:64826 2023-04-08 02:54:42,068 DEBUG [Thread-992] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:54:42,081 DEBUG [Thread-1000] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x210d1008 to 127.0.0.1:64826 2023-04-08 02:54:42,081 DEBUG [Thread-1000] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:54:42,081 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-08 02:54:42,081 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 30 2023-04-08 02:54:42,081 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 78 2023-04-08 02:54:42,081 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 33 2023-04-08 02:54:42,081 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 141 2023-04-08 02:54:42,081 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 29 2023-04-08 02:54:42,081 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-08 02:54:42,081 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(398): read 5525 2023-04-08 02:54:42,081 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(398): read 5264 2023-04-08 02:54:42,081 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-08 02:54:42,081 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(402): scanned 2283 2023-04-08 02:54:42,081 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(403): verified 6849 rows 2023-04-08 02:54:42,081 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(402): scanned 2276 2023-04-08 02:54:42,081 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(403): verified 6820 rows 2023-04-08 02:54:42,081 INFO [Listener at localhost/39741] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-08 02:54:42,081 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x52274650 to 127.0.0.1:64826 2023-04-08 02:54:42,081 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:54:42,083 INFO [Listener at localhost/39741] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-08 02:54:42,084 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-08 02:54:42,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure2.ProcedureExecutor(1029): Stored pid=24, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-08 02:54:42,088 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-08 02:54:42,092 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680922482092"}]},"ts":"1680922482092"} 2023-04-08 02:54:42,093 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-08 02:54:42,095 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-08 02:54:42,095 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d38b4e710235c58cc9479560817e4c8d, UNASSIGN}] 2023-04-08 02:54:42,097 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=25, ppid=24, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d38b4e710235c58cc9479560817e4c8d, UNASSIGN 2023-04-08 02:54:42,097 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=d38b4e710235c58cc9479560817e4c8d, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:42,098 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922482097"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680922482097"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680922482097"}]},"ts":"1680922482097"} 2023-04-08 02:54:42,098 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=26, ppid=25, state=RUNNABLE; CloseRegionProcedure d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983}] 2023-04-08 02:54:42,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-08 02:54:42,250 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:42,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing d38b4e710235c58cc9479560817e4c8d, disabling compactions & flushes 2023-04-08 02:54:42,250 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:42,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:42,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. after waiting 0 ms 2023-04-08 02:54:42,250 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:42,250 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing d38b4e710235c58cc9479560817e4c8d 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-08 02:54:42,251 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=A 2023-04-08 02:54:42,251 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:42,251 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=B 2023-04-08 02:54:42,251 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:42,251 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d38b4e710235c58cc9479560817e4c8d, store=C 2023-04-08 02:54:42,251 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:42,390 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-08 02:54:42,667 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304086a2deae675b7468f83c775f283f48d2f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304086a2deae675b7468f83c775f283f48d2f 2023-04-08 02:54:42,691 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-08 02:54:43,072 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=390, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/1b02a17387684cd4a289e4d058656768 2023-04-08 02:54:43,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-08 02:54:43,482 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/2f9d621ee77d4d379911e73887074192 2023-04-08 02:54:43,894 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=390 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/a458ce13ba5541e68b8f3436e27288a2 2023-04-08 02:54:43,898 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/A/1b02a17387684cd4a289e4d058656768 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/1b02a17387684cd4a289e4d058656768 2023-04-08 02:54:43,902 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/1b02a17387684cd4a289e4d058656768, entries=100, sequenceid=390, filesize=18.6 K 2023-04-08 02:54:43,903 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/B/2f9d621ee77d4d379911e73887074192 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/2f9d621ee77d4d379911e73887074192 2023-04-08 02:54:43,907 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/2f9d621ee77d4d379911e73887074192, entries=100, sequenceid=390, filesize=9.5 K 2023-04-08 02:54:43,908 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/.tmp/C/a458ce13ba5541e68b8f3436e27288a2 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a458ce13ba5541e68b8f3436e27288a2 2023-04-08 02:54:43,913 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a458ce13ba5541e68b8f3436e27288a2, entries=100, sequenceid=390, filesize=9.5 K 2023-04-08 02:54:43,913 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for d38b4e710235c58cc9479560817e4c8d in 1663ms, sequenceid=390, compaction requested=true 2023-04-08 02:54:43,936 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/080a10550c5a417e880d276fa08a8dbe, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/c6f6445e4bf14b2f9b7e03c4c4bbf616, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/7a385dc63ae64b84bf98a5d9bdac19c0, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/bf551fae7fab470392046441b9f229df, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/b872522a3af647b4aa4d477f5b7cb4f4, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/c25e5fecc86b45c49ca41cba83c61cab, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/3b24d5cc7faf403f89d3af951aabc72c, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/2f944dee9e014e02afe5de39d15e5389, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/720e931a46dd4c39abcad436db3eb05a, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/e9f8c45ddb6449a38207a6d70e666332, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/338061bf97124e4d98b960781cb9a383, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/03da205c9283405592309a68adeb2667, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/ebf35745d2c44542be96b3cb82776b66, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/e24684b511a04dfe997c5e72a6cf51a8, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/22da8d5705404c668c64df3da840d24f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/ba40e967b6fa4f12b7c8b4c4ce277a0f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/4c88f2a65c5447339fc931212a3a36e5, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/d6f00df893964b55939916c8d647edd2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/d37cfa71dbae4976bf757589dca25950, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/ed3dd768c5304e98a7e22e0111fc3e29, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/69ea031980564edd8934cd77d6d2f990, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/e26f12580f664201ae043a244d66ba77] to archive 2023-04-08 02:54:43,938 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 02:54:43,940 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/080a10550c5a417e880d276fa08a8dbe to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/080a10550c5a417e880d276fa08a8dbe 2023-04-08 02:54:43,941 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/c6f6445e4bf14b2f9b7e03c4c4bbf616 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/c6f6445e4bf14b2f9b7e03c4c4bbf616 2023-04-08 02:54:43,942 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/7a385dc63ae64b84bf98a5d9bdac19c0 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/7a385dc63ae64b84bf98a5d9bdac19c0 2023-04-08 02:54:43,943 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/bf551fae7fab470392046441b9f229df to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/bf551fae7fab470392046441b9f229df 2023-04-08 02:54:43,944 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/b872522a3af647b4aa4d477f5b7cb4f4 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/b872522a3af647b4aa4d477f5b7cb4f4 2023-04-08 02:54:43,945 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/c25e5fecc86b45c49ca41cba83c61cab to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/c25e5fecc86b45c49ca41cba83c61cab 2023-04-08 02:54:43,946 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/3b24d5cc7faf403f89d3af951aabc72c to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/3b24d5cc7faf403f89d3af951aabc72c 2023-04-08 02:54:43,947 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/2f944dee9e014e02afe5de39d15e5389 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/2f944dee9e014e02afe5de39d15e5389 2023-04-08 02:54:43,948 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/720e931a46dd4c39abcad436db3eb05a to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/720e931a46dd4c39abcad436db3eb05a 2023-04-08 02:54:43,949 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/e9f8c45ddb6449a38207a6d70e666332 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/e9f8c45ddb6449a38207a6d70e666332 2023-04-08 02:54:43,950 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/338061bf97124e4d98b960781cb9a383 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/338061bf97124e4d98b960781cb9a383 2023-04-08 02:54:43,951 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/03da205c9283405592309a68adeb2667 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/03da205c9283405592309a68adeb2667 2023-04-08 02:54:43,952 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/ebf35745d2c44542be96b3cb82776b66 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/ebf35745d2c44542be96b3cb82776b66 2023-04-08 02:54:43,953 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/e24684b511a04dfe997c5e72a6cf51a8 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/e24684b511a04dfe997c5e72a6cf51a8 2023-04-08 02:54:43,954 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/22da8d5705404c668c64df3da840d24f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/22da8d5705404c668c64df3da840d24f 2023-04-08 02:54:43,955 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/ba40e967b6fa4f12b7c8b4c4ce277a0f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/ba40e967b6fa4f12b7c8b4c4ce277a0f 2023-04-08 02:54:43,955 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/4c88f2a65c5447339fc931212a3a36e5 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/4c88f2a65c5447339fc931212a3a36e5 2023-04-08 02:54:43,957 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/d6f00df893964b55939916c8d647edd2 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/d6f00df893964b55939916c8d647edd2 2023-04-08 02:54:43,958 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/d37cfa71dbae4976bf757589dca25950 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/d37cfa71dbae4976bf757589dca25950 2023-04-08 02:54:43,958 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/ed3dd768c5304e98a7e22e0111fc3e29 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/ed3dd768c5304e98a7e22e0111fc3e29 2023-04-08 02:54:43,959 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/69ea031980564edd8934cd77d6d2f990 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/69ea031980564edd8934cd77d6d2f990 2023-04-08 02:54:43,960 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/e26f12580f664201ae043a244d66ba77 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/e26f12580f664201ae043a244d66ba77 2023-04-08 02:54:43,962 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/72811cc7722144bab7f684d1cad2be1b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/97d505898f1f442c972b27c3a7badb61, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/b0c4c7ca8e164cefa8669b107dedc29f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/4668bb62f81d4421ac705a03356486a9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/d095085398d640208fff3f508aa78d31, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/3559e3e1c5114d78b7e4bca509d062db, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/eb548040fef54907bd18a9e0eaef4424, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/66855a5eb08041cebf8d309d1bace3ef, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/0d3ce136a3b245f181728b3cdb6223c9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/668ef7a7188e4a50b4f770dc4c1d5a07, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/b0e1ac90b93a402b9a5cbe57b9d90cc7, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/6136115768c44833a7295f54f914420d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/38475076b31743548fb0adb564fde52f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/68ca387b786446a1978de536f129462d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/45c20656a8f54cdaa3e1338e0a0dc529, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/173b9070f3194e42916eb91d7ef8dbec, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/fd5cea4cdb314be98a07f92f6178f792, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/772cdd6afb064d0cb4bd1270a36a8baf, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/a0c0d0013445410faf57e8697eb0d9f4, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/920418ad8e434abab51f0ab0a6991687, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/c1c8b982cb0a43dcbfb90dd484852dd6, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/0d8dcaceb83c48659be94d04ed62413a] to archive 2023-04-08 02:54:43,963 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 02:54:43,964 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/72811cc7722144bab7f684d1cad2be1b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/72811cc7722144bab7f684d1cad2be1b 2023-04-08 02:54:43,965 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/97d505898f1f442c972b27c3a7badb61 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/97d505898f1f442c972b27c3a7badb61 2023-04-08 02:54:43,966 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/b0c4c7ca8e164cefa8669b107dedc29f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/b0c4c7ca8e164cefa8669b107dedc29f 2023-04-08 02:54:43,967 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/4668bb62f81d4421ac705a03356486a9 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/4668bb62f81d4421ac705a03356486a9 2023-04-08 02:54:43,968 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/d095085398d640208fff3f508aa78d31 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/d095085398d640208fff3f508aa78d31 2023-04-08 02:54:43,969 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/3559e3e1c5114d78b7e4bca509d062db to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/3559e3e1c5114d78b7e4bca509d062db 2023-04-08 02:54:43,970 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/eb548040fef54907bd18a9e0eaef4424 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/eb548040fef54907bd18a9e0eaef4424 2023-04-08 02:54:43,971 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/66855a5eb08041cebf8d309d1bace3ef to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/66855a5eb08041cebf8d309d1bace3ef 2023-04-08 02:54:43,972 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/0d3ce136a3b245f181728b3cdb6223c9 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/0d3ce136a3b245f181728b3cdb6223c9 2023-04-08 02:54:43,973 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/668ef7a7188e4a50b4f770dc4c1d5a07 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/668ef7a7188e4a50b4f770dc4c1d5a07 2023-04-08 02:54:43,973 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/b0e1ac90b93a402b9a5cbe57b9d90cc7 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/b0e1ac90b93a402b9a5cbe57b9d90cc7 2023-04-08 02:54:43,974 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/6136115768c44833a7295f54f914420d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/6136115768c44833a7295f54f914420d 2023-04-08 02:54:43,975 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/38475076b31743548fb0adb564fde52f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/38475076b31743548fb0adb564fde52f 2023-04-08 02:54:43,976 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/68ca387b786446a1978de536f129462d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/68ca387b786446a1978de536f129462d 2023-04-08 02:54:43,977 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/45c20656a8f54cdaa3e1338e0a0dc529 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/45c20656a8f54cdaa3e1338e0a0dc529 2023-04-08 02:54:43,978 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/173b9070f3194e42916eb91d7ef8dbec to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/173b9070f3194e42916eb91d7ef8dbec 2023-04-08 02:54:43,979 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/fd5cea4cdb314be98a07f92f6178f792 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/fd5cea4cdb314be98a07f92f6178f792 2023-04-08 02:54:43,980 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/772cdd6afb064d0cb4bd1270a36a8baf to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/772cdd6afb064d0cb4bd1270a36a8baf 2023-04-08 02:54:43,981 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/a0c0d0013445410faf57e8697eb0d9f4 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/a0c0d0013445410faf57e8697eb0d9f4 2023-04-08 02:54:43,982 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/920418ad8e434abab51f0ab0a6991687 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/920418ad8e434abab51f0ab0a6991687 2023-04-08 02:54:43,983 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/c1c8b982cb0a43dcbfb90dd484852dd6 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/c1c8b982cb0a43dcbfb90dd484852dd6 2023-04-08 02:54:43,984 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/0d8dcaceb83c48659be94d04ed62413a to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/0d8dcaceb83c48659be94d04ed62413a 2023-04-08 02:54:43,986 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/0ff8bedb3705457fbe57b88601b98914, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/0bc82dc108204b4ba933b984799b2a92, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/9d2a641bfa4e45d6b67794b45a66e464, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/1c999831d17c498b93098e5b4aa8c229, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a3a95cfe2de949ccba1f9d5a3b00ea71, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/521c0c2e2ded4588ae29ab832594555a, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/794438b911d54527b31701c92d7cec5b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/cdecaa039d904ae79ef831ba5ac06485, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/50fd46b457b24105b9b975fb761a8d9c, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/b13ccba2c64c47a88c8ed870de50d938, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/6141f9392d684adea47326950dd4ec5d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/6853699a53bd4f1e92920fbb571aa1e2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/031d7681eb334ee698f82c114ec9a647, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a7daa1f6c6df4312980000d8e769dbd7, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/5598d76e6927482d923eb4b12d909e5b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a862ed86fed941189c3127c6ae737109, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/efb9ca50a2da407aaf37464021164aba, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/7ed1462a85d147e99fa87ad4a3585454, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/c92700639dc649b19d57b17bd81b29ea, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/daa0654f7fb7410e91e48c47eacc9c53, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/1b08a62ecda2431da50465118bb80ae3, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a045c4aa94744b8fbc5b1ae14e7495d1] to archive 2023-04-08 02:54:43,987 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 02:54:43,988 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/0ff8bedb3705457fbe57b88601b98914 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/0ff8bedb3705457fbe57b88601b98914 2023-04-08 02:54:43,989 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/0bc82dc108204b4ba933b984799b2a92 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/0bc82dc108204b4ba933b984799b2a92 2023-04-08 02:54:43,989 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/9d2a641bfa4e45d6b67794b45a66e464 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/9d2a641bfa4e45d6b67794b45a66e464 2023-04-08 02:54:43,990 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/1c999831d17c498b93098e5b4aa8c229 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/1c999831d17c498b93098e5b4aa8c229 2023-04-08 02:54:43,991 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a3a95cfe2de949ccba1f9d5a3b00ea71 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a3a95cfe2de949ccba1f9d5a3b00ea71 2023-04-08 02:54:43,992 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/521c0c2e2ded4588ae29ab832594555a to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/521c0c2e2ded4588ae29ab832594555a 2023-04-08 02:54:43,993 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/794438b911d54527b31701c92d7cec5b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/794438b911d54527b31701c92d7cec5b 2023-04-08 02:54:43,994 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/cdecaa039d904ae79ef831ba5ac06485 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/cdecaa039d904ae79ef831ba5ac06485 2023-04-08 02:54:43,995 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/50fd46b457b24105b9b975fb761a8d9c to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/50fd46b457b24105b9b975fb761a8d9c 2023-04-08 02:54:43,996 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/b13ccba2c64c47a88c8ed870de50d938 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/b13ccba2c64c47a88c8ed870de50d938 2023-04-08 02:54:43,997 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/6141f9392d684adea47326950dd4ec5d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/6141f9392d684adea47326950dd4ec5d 2023-04-08 02:54:43,997 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/6853699a53bd4f1e92920fbb571aa1e2 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/6853699a53bd4f1e92920fbb571aa1e2 2023-04-08 02:54:43,998 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/031d7681eb334ee698f82c114ec9a647 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/031d7681eb334ee698f82c114ec9a647 2023-04-08 02:54:43,999 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a7daa1f6c6df4312980000d8e769dbd7 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a7daa1f6c6df4312980000d8e769dbd7 2023-04-08 02:54:44,000 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/5598d76e6927482d923eb4b12d909e5b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/5598d76e6927482d923eb4b12d909e5b 2023-04-08 02:54:44,001 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a862ed86fed941189c3127c6ae737109 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a862ed86fed941189c3127c6ae737109 2023-04-08 02:54:44,002 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/efb9ca50a2da407aaf37464021164aba to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/efb9ca50a2da407aaf37464021164aba 2023-04-08 02:54:44,003 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/7ed1462a85d147e99fa87ad4a3585454 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/7ed1462a85d147e99fa87ad4a3585454 2023-04-08 02:54:44,003 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/c92700639dc649b19d57b17bd81b29ea to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/c92700639dc649b19d57b17bd81b29ea 2023-04-08 02:54:44,004 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/daa0654f7fb7410e91e48c47eacc9c53 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/daa0654f7fb7410e91e48c47eacc9c53 2023-04-08 02:54:44,005 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/1b08a62ecda2431da50465118bb80ae3 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/1b08a62ecda2431da50465118bb80ae3 2023-04-08 02:54:44,006 DEBUG [StoreCloser-TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a045c4aa94744b8fbc5b1ae14e7495d1 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a045c4aa94744b8fbc5b1ae14e7495d1 2023-04-08 02:54:44,011 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/recovered.edits/393.seqid, newMaxSeqId=393, maxSeqId=4 2023-04-08 02:54:44,013 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d. 2023-04-08 02:54:44,013 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for d38b4e710235c58cc9479560817e4c8d: 2023-04-08 02:54:44,014 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:44,015 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=25 updating hbase:meta row=d38b4e710235c58cc9479560817e4c8d, regionState=CLOSED 2023-04-08 02:54:44,015 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922484015"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680922484015"}]},"ts":"1680922484015"} 2023-04-08 02:54:44,017 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=26, resume processing ppid=25 2023-04-08 02:54:44,017 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=26, ppid=25, state=SUCCESS; CloseRegionProcedure d38b4e710235c58cc9479560817e4c8d, server=jenkins-hbase4.apache.org,35089,1680922417983 in 1.9170 sec 2023-04-08 02:54:44,018 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=25, resume processing ppid=24 2023-04-08 02:54:44,019 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=25, ppid=24, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d38b4e710235c58cc9479560817e4c8d, UNASSIGN in 1.9220 sec 2023-04-08 02:54:44,019 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680922484019"}]},"ts":"1680922484019"} 2023-04-08 02:54:44,020 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-08 02:54:44,021 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-08 02:54:44,022 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=24, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9380 sec 2023-04-08 02:54:44,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=24 2023-04-08 02:54:44,193 INFO [Listener at localhost/39741] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 24 completed 2023-04-08 02:54:44,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-08 02:54:44,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure2.ProcedureExecutor(1029): Stored pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:54:44,196 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=27, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:54:44,197 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-08 02:54:44,197 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=27, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:54:44,202 DEBUG [HFileArchiver-5] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:44,207 DEBUG [HFileArchiver-5] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A, FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B, FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C, FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/recovered.edits] 2023-04-08 02:54:44,211 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/1b02a17387684cd4a289e4d058656768 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/1b02a17387684cd4a289e4d058656768 2023-04-08 02:54:44,212 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/6a074ef96fb44dfab4433cf930735d97 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/6a074ef96fb44dfab4433cf930735d97 2023-04-08 02:54:44,214 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/8a033f311b804ef98fb4009771d0f9df to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/A/8a033f311b804ef98fb4009771d0f9df 2023-04-08 02:54:44,217 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/18fa9bbad11c4d8199adad343ff44549 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/18fa9bbad11c4d8199adad343ff44549 2023-04-08 02:54:44,218 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/2f9d621ee77d4d379911e73887074192 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/2f9d621ee77d4d379911e73887074192 2023-04-08 02:54:44,219 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/d6be5bcff0074e3f8c087e01974c7c8f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/B/d6be5bcff0074e3f8c087e01974c7c8f 2023-04-08 02:54:44,222 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/17eefddb98f04f2c8bda955733bdaa15 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/17eefddb98f04f2c8bda955733bdaa15 2023-04-08 02:54:44,223 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/414507a808ec4e35a046fb503d3dea0d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/414507a808ec4e35a046fb503d3dea0d 2023-04-08 02:54:44,225 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a458ce13ba5541e68b8f3436e27288a2 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/C/a458ce13ba5541e68b8f3436e27288a2 2023-04-08 02:54:44,228 DEBUG [HFileArchiver-5] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/recovered.edits/393.seqid to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d/recovered.edits/393.seqid 2023-04-08 02:54:44,230 DEBUG [HFileArchiver-5] backup.HFileArchiver(596): Deleted hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d38b4e710235c58cc9479560817e4c8d 2023-04-08 02:54:44,230 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-08 02:54:44,230 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-08 02:54:44,240 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-08 02:54:44,245 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040812546a5c68b440d3ba9c785fd25a69e7 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040812546a5c68b440d3ba9c785fd25a69e7 2023-04-08 02:54:44,247 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040822746b48639d4e368190497c00f2a962 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040822746b48639d4e368190497c00f2a962 2023-04-08 02:54:44,248 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408263e64a5020c432ea56f5ff810552370 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408263e64a5020c432ea56f5ff810552370 2023-04-08 02:54:44,250 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408290aaf314be044ee84cc8b2ba00de88f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408290aaf314be044ee84cc8b2ba00de88f 2023-04-08 02:54:44,252 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304082c07a0f7ef584ee2a27cacfde9b15d6a to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304082c07a0f7ef584ee2a27cacfde9b15d6a 2023-04-08 02:54:44,254 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040850821e081e844073ae40664a646fa8c4 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040850821e081e844073ae40664a646fa8c4 2023-04-08 02:54:44,256 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040858239c9a5fc641abb3570b8c54eac5e4 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040858239c9a5fc641abb3570b8c54eac5e4 2023-04-08 02:54:44,258 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304085a0f8334798d4a188421e989c11fa9e2 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304085a0f8334798d4a188421e989c11fa9e2 2023-04-08 02:54:44,259 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304086a2deae675b7468f83c775f283f48d2f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304086a2deae675b7468f83c775f283f48d2f 2023-04-08 02:54:44,261 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304087e71adece28844718ebe22b519492f8b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304087e71adece28844718ebe22b519492f8b 2023-04-08 02:54:44,262 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408920a7ecc42da4c85adcca4cb06584182 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408920a7ecc42da4c85adcca4cb06584182 2023-04-08 02:54:44,264 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408979daf36b0b148108e4872c00c64f912 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408979daf36b0b148108e4872c00c64f912 2023-04-08 02:54:44,265 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304089d56cb3c86b2419ca74d66abf849103e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304089d56cb3c86b2419ca74d66abf849103e 2023-04-08 02:54:44,267 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408a03c3cc5830c4dedb349a6c669590b16 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408a03c3cc5830c4dedb349a6c669590b16 2023-04-08 02:54:44,268 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408a5b7f57defa7432a915176ef0f3ae633 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408a5b7f57defa7432a915176ef0f3ae633 2023-04-08 02:54:44,270 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408c64fb392d5f34646ae75fded3d5d0d64 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408c64fb392d5f34646ae75fded3d5d0d64 2023-04-08 02:54:44,271 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408cfa027c3f819483e9fcf472a40f28668 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408cfa027c3f819483e9fcf472a40f28668 2023-04-08 02:54:44,272 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408d09f07d22330422c8ec752f4871d265b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408d09f07d22330422c8ec752f4871d265b 2023-04-08 02:54:44,273 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408df0248050459447d8d4c2d47a45e7fce to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408df0248050459447d8d4c2d47a45e7fce 2023-04-08 02:54:44,275 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408e1d8dd7e6d9942a4a7fc4fee111ff236 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408e1d8dd7e6d9942a4a7fc4fee111ff236 2023-04-08 02:54:44,277 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408ee72836b52a547f197a3631e87d0405d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408ee72836b52a547f197a3631e87d0405d 2023-04-08 02:54:44,277 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-08 02:54:44,280 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=27, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:54:44,282 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-08 02:54:44,284 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-08 02:54:44,285 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=27, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:54:44,285 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-08 02:54:44,286 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680922484285"}]},"ts":"9223372036854775807"} 2023-04-08 02:54:44,289 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-08 02:54:44,289 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => d38b4e710235c58cc9479560817e4c8d, NAME => 'TestAcidGuarantees,,1680922451228.d38b4e710235c58cc9479560817e4c8d.', STARTKEY => '', ENDKEY => ''}] 2023-04-08 02:54:44,289 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-08 02:54:44,290 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680922484290"}]},"ts":"9223372036854775807"} 2023-04-08 02:54:44,297 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-08 02:54:44,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-08 02:54:44,299 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=27, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:54:44,301 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=27, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 105 msec 2023-04-08 02:54:44,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=27 2023-04-08 02:54:44,499 INFO [Listener at localhost/39741] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 27 completed 2023-04-08 02:54:44,511 INFO [Listener at localhost/39741] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobMixedAtomicity Thread=224 (was 233), OpenFileDescriptor=534 (was 569), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=363 (was 424), ProcessCount=170 (was 170), AvailableMemoryMB=1634 (was 1789) 2023-04-08 02:54:44,522 INFO [Listener at localhost/39741] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=224, OpenFileDescriptor=534, MaxFileDescriptor=60000, SystemLoadAverage=363, ProcessCount=170, AvailableMemoryMB=1633 2023-04-08 02:54:44,524 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-08 02:54:44,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-08 02:54:44,525 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure2.ProcedureExecutor(1029): Stored pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-08 02:54:44,526 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-08 02:54:44,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 28 2023-04-08 02:54:44,527 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-08 02:54:44,527 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-08 02:54:44,528 DEBUG [HFileArchiver-6] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:44,529 DEBUG [HFileArchiver-6] backup.HFileArchiver(153): Directory hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850 empty. 2023-04-08 02:54:44,529 DEBUG [HFileArchiver-6] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:44,529 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-08 02:54:44,536 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-08 02:54:44,538 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => d3e62a2534068dd21a6b490550cac850, NAME => 'TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp 2023-04-08 02:54:44,628 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-08 02:54:44,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-08 02:54:44,943 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 02:54:44,943 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing d3e62a2534068dd21a6b490550cac850, disabling compactions & flushes 2023-04-08 02:54:44,943 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:44,943 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:44,943 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. after waiting 0 ms 2023-04-08 02:54:44,943 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:44,943 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:44,944 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:44,946 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-08 02:54:44,947 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922484946"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680922484946"}]},"ts":"1680922484946"} 2023-04-08 02:54:44,948 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-08 02:54:44,948 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-08 02:54:44,948 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680922484948"}]},"ts":"1680922484948"} 2023-04-08 02:54:44,949 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-08 02:54:44,953 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d3e62a2534068dd21a6b490550cac850, ASSIGN}] 2023-04-08 02:54:44,954 INFO [PEWorker-3] procedure.MasterProcedureScheduler(727): Took xlock for pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d3e62a2534068dd21a6b490550cac850, ASSIGN 2023-04-08 02:54:44,955 INFO [PEWorker-3] assignment.TransitRegionStateProcedure(193): Starting pid=29, ppid=28, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=d3e62a2534068dd21a6b490550cac850, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35089,1680922417983; forceNewPlan=false, retain=false 2023-04-08 02:54:45,106 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=d3e62a2534068dd21a6b490550cac850, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,106 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922485106"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680922485106"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680922485106"}]},"ts":"1680922485106"} 2023-04-08 02:54:45,107 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=30, ppid=29, state=RUNNABLE; OpenRegionProcedure d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983}] 2023-04-08 02:54:45,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-08 02:54:45,266 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:45,266 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => d3e62a2534068dd21a6b490550cac850, NAME => 'TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.', STARTKEY => '', ENDKEY => ''} 2023-04-08 02:54:45,266 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:45,267 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 02:54:45,267 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:45,267 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:45,268 INFO [StoreOpener-d3e62a2534068dd21a6b490550cac850-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:45,270 DEBUG [StoreOpener-d3e62a2534068dd21a6b490550cac850-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A 2023-04-08 02:54:45,270 DEBUG [StoreOpener-d3e62a2534068dd21a6b490550cac850-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A 2023-04-08 02:54:45,270 INFO [StoreOpener-d3e62a2534068dd21a6b490550cac850-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 02:54:45,270 INFO [StoreOpener-d3e62a2534068dd21a6b490550cac850-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d3e62a2534068dd21a6b490550cac850 columnFamilyName A 2023-04-08 02:54:45,271 INFO [StoreOpener-d3e62a2534068dd21a6b490550cac850-1] regionserver.HStore(310): Store=d3e62a2534068dd21a6b490550cac850/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:54:45,271 INFO [StoreOpener-d3e62a2534068dd21a6b490550cac850-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:45,273 DEBUG [StoreOpener-d3e62a2534068dd21a6b490550cac850-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B 2023-04-08 02:54:45,273 DEBUG [StoreOpener-d3e62a2534068dd21a6b490550cac850-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B 2023-04-08 02:54:45,273 INFO [StoreOpener-d3e62a2534068dd21a6b490550cac850-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 02:54:45,273 INFO [StoreOpener-d3e62a2534068dd21a6b490550cac850-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d3e62a2534068dd21a6b490550cac850 columnFamilyName B 2023-04-08 02:54:45,274 INFO [StoreOpener-d3e62a2534068dd21a6b490550cac850-1] regionserver.HStore(310): Store=d3e62a2534068dd21a6b490550cac850/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:54:45,274 INFO [StoreOpener-d3e62a2534068dd21a6b490550cac850-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:45,276 DEBUG [StoreOpener-d3e62a2534068dd21a6b490550cac850-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C 2023-04-08 02:54:45,276 DEBUG [StoreOpener-d3e62a2534068dd21a6b490550cac850-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C 2023-04-08 02:54:45,276 INFO [StoreOpener-d3e62a2534068dd21a6b490550cac850-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 02:54:45,276 INFO [StoreOpener-d3e62a2534068dd21a6b490550cac850-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region d3e62a2534068dd21a6b490550cac850 columnFamilyName C 2023-04-08 02:54:45,277 INFO [StoreOpener-d3e62a2534068dd21a6b490550cac850-1] regionserver.HStore(310): Store=d3e62a2534068dd21a6b490550cac850/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:54:45,277 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:45,277 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:45,278 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:45,279 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-08 02:54:45,280 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:45,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-08 02:54:45,282 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened d3e62a2534068dd21a6b490550cac850; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=72177145, jitterRate=0.0755232721567154}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-08 02:54:45,282 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:45,283 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., pid=30, masterSystemTime=1680922485259 2023-04-08 02:54:45,284 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:45,284 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:45,284 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=29 updating hbase:meta row=d3e62a2534068dd21a6b490550cac850, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,285 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922485284"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680922485284"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680922485284"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680922485284"}]},"ts":"1680922485284"} 2023-04-08 02:54:45,287 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=30, resume processing ppid=29 2023-04-08 02:54:45,287 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=30, ppid=29, state=SUCCESS; OpenRegionProcedure d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 in 178 msec 2023-04-08 02:54:45,288 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=29, resume processing ppid=28 2023-04-08 02:54:45,288 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=29, ppid=28, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d3e62a2534068dd21a6b490550cac850, ASSIGN in 335 msec 2023-04-08 02:54:45,289 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-08 02:54:45,289 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680922485289"}]},"ts":"1680922485289"} 2023-04-08 02:54:45,290 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-08 02:54:45,291 INFO [PEWorker-3] procedure.CreateTableProcedure(80): pid=28, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-08 02:54:45,293 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=28, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 768 msec 2023-04-08 02:54:45,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=28 2023-04-08 02:54:45,631 INFO [Listener at localhost/39741] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 28 completed 2023-04-08 02:54:45,632 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x12e82edd to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:54:45,636 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7ed1dde9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:54:45,638 DEBUG [Listener at localhost/39741] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:54:45,639 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53086, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:54:45,640 DEBUG [Listener at localhost/39741] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-08 02:54:45,642 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53150, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-08 02:54:45,644 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x23c97092 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:54:45,649 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70043cf0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:54:45,649 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x5c124820 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:54:45,653 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@34b7fed0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:54:45,653 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x3157d4df to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:54:45,657 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7f4ad99d, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:54:45,657 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x3cf5bdb7 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:54:45,662 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@330b77da, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:54:45,662 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x6919b2a0 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:54:45,667 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@7810a231, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:54:45,667 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x08cd7cb0 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:54:45,671 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@753156b2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:54:45,671 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x0ea1ca03 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:54:45,676 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6dea0f9b, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:54:45,676 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x433f251d to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:54:45,681 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@434e0eab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:54:45,681 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x66ee0bd0 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:54:45,688 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5c43dddd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:54:45,688 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x3b4658f8 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:54:45,691 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2de064cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:54:45,696 DEBUG [hconnection-0x7169b22e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:54:45,696 DEBUG [hconnection-0x2b59d60a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:54:45,696 DEBUG [hconnection-0x16674b36-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:54:45,697 DEBUG [hconnection-0x42b68ec2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:54:45,697 DEBUG [hconnection-0x774150a2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:54:45,697 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:45,697 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53098, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:54:45,698 DEBUG [hconnection-0x3599612a-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:54:45,698 DEBUG [hconnection-0x44c79d72-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:54:45,701 DEBUG [hconnection-0xb8e8661-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:54:45,701 DEBUG [hconnection-0x46c65493-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:54:45,701 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53104, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:54:45,701 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53102, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:54:45,701 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53142, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:54:45,701 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53126, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:54:45,701 DEBUG [hconnection-0x4cd1cdaa-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:54:45,701 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53122, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:54:45,703 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53178, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:54:45,706 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53158, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:54:45,708 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53172, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:54:45,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:45,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:45,713 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:45,714 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:53180, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:54:45,714 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:45,714 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:45,714 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:45,715 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:45,715 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:45,720 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:45,720 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,721 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:45,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:45,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:45,722 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:45,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:45,727 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-08 02:54:45,727 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,727 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:45,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:45,727 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:45,728 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:54:45,728 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:45,728 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:45,728 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:45,728 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:54:45,728 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:45,728 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:45,728 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:45,729 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:45,729 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:54:45,729 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:45,729 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:45,729 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:45,729 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:45,729 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:45,729 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:45,729 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:45,729 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:45,729 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:45,729 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,729 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:45,729 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:45,731 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,731 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:45,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:45,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:45,731 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:45,731 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:45,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:45,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:45,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:45,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:45,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:45,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6ac6292c[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:45,733 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:45,733 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:45,735 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:45,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:45,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:45,735 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,736 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:45,736 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:45,736 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:45,736 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:45,737 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:45,737 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:45,737 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:45,739 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:45,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:45,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:45,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:45,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:45,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:45,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:45,741 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/21df7a61da0d40f88f0158b0ed3f0876 2023-04-08 02:54:45,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:45,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:45,746 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:45,747 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:45,747 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:45,747 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:45,747 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:45,748 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:45,748 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:45,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:45,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:45,748 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:45,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:45,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:45,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:45,749 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:45,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:45,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:45,749 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:45,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:45,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:45,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:45,751 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:45,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:45,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,754 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:45,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922545748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,754 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:45,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922545748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,755 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:45,755 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:45,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:45,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:45,755 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,755 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:45,755 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:45,755 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,755 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:45,755 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:45,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922545749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,755 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:45,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:45,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:45,755 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:45,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:45,756 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:45,756 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:45,756 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:45,756 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:45,756 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:45,756 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:45,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922545751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922545751, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,856 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:45,857 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:45,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:45,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922545856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:45,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922545856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,857 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:45,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922545857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,858 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:45,858 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:45,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922545857, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,859 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:45,859 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922545858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:45,860 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:45,860 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:45,860 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:45,860 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:45,860 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:45,860 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:45,861 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:45,863 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:45,863 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:45,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:45,863 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:45,863 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,863 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:45,863 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:45,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:45,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:45,864 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:45,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:45,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:45,864 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:45,864 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:45,864 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:45,864 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:45,864 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:45,865 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:45,865 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:45,865 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:45,865 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:45,865 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:45,865 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:45,865 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,865 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:45,865 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:45,868 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,868 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:45,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:45,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:45,868 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:45,868 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:45,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:45,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:45,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:45,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:45,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:45,869 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@41587739[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:45,869 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:45,869 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:45,870 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:45,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:45,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:45,870 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:45,871 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:45,871 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:45,871 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,871 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:45,876 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:45,876 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:45,876 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,876 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:45,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:45,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:45,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:45,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:45,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:45,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:45,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:45,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:45,879 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:45,879 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:45,879 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:45,879 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:45,879 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:45,879 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:45,881 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:45,881 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:45,881 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:45,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:45,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:45,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:45,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:45,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:45,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:45,883 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:45,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:45,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:45,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:45,884 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:45,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:45,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:45,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,888 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:45,888 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:45,888 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,888 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:45,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:45,889 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:45,888 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:45,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:45,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:45,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:45,889 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:45,889 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:45,889 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:45,889 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:45,889 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:45,889 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:45,889 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:45,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:45,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:45,989 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:45,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:45,997 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:45,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:45,998 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:45,999 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:45,999 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:45,999 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:45,999 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:45,999 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:45,999 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,001 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:46,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:46,001 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,002 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,002 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:46,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:46,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,003 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:46,004 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:46,004 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:46,004 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:46,004 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:46,004 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:46,007 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:46,007 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:46,008 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:46,008 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:46,008 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:46,008 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:46,008 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:46,009 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:46,009 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,009 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:46,009 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:46,011 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,011 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:46,012 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:46,012 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:46,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:46,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:46,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@f8f5f4b[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:46,014 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:46,014 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,015 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,016 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:46,016 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:46,016 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:46,016 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,016 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:46,017 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,017 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:46,017 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:46,017 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:46,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:46,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:46,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:46,020 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:46,020 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:46,020 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:46,020 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:46,020 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:46,020 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:46,022 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:46,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:46,022 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,022 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:46,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:46,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:46,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:46,023 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:46,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,024 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:46,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,032 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:46,032 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:46,032 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,032 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:46,032 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:46,032 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:46,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:46,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:46,032 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,032 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,032 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,032 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,033 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,033 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:46,033 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:46,033 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:46,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:46,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:46,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922546059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:46,060 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922546059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,060 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:46,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922546059, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,061 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:46,061 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922546060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,062 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:46,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922546060, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,133 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:46,133 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:46,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:46,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:46,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:46,136 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:46,136 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:46,136 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:46,136 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:46,136 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,137 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,138 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:46,138 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:46,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:46,138 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,138 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:46,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,138 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:46,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,139 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:46,139 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:46,139 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:46,139 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:46,139 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:46,139 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:46,140 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:46,140 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:46,140 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:46,140 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:46,140 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:46,140 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:46,140 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:46,140 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:46,140 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,140 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:46,140 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:46,141 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,141 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:46,142 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,142 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:46,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:46,143 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:46,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4e898b70[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:46,143 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,144 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,144 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,144 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:46,144 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:46,144 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,145 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:46,144 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:46,147 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,147 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:46,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:46,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:46,147 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:46,147 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:46,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:46,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:46,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:46,149 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:46,149 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:46,150 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:46,150 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:46,150 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:46,150 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:46,151 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,151 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:46,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:46,151 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:46,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,152 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,152 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:46,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:46,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:46,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:46,153 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:46,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,155 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,155 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,155 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:46,155 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:46,155 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:46,156 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:46,156 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:46,156 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:46,155 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,156 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:46,155 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:46,156 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:46,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:46,156 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,156 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:46,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:46,210 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/ec2af5492feb49128a938cc0c35d349c 2023-04-08 02:54:46,256 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:46,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:46,258 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:46,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:46,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:46,259 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:46,259 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:46,259 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:46,259 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:46,260 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,260 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,261 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:46,261 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,261 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:46,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:46,261 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,261 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:46,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,262 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:46,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,262 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:46,262 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:46,262 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:46,263 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:46,263 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:46,263 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:46,263 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:46,263 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:46,263 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:46,263 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:46,263 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:46,263 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:46,263 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:46,263 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:46,263 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,263 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:46,263 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:46,265 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,265 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:46,266 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,266 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:46,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:46,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:46,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:46,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@155d44c4[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:46,267 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:46,267 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,268 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,269 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:46,269 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:46,269 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,269 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:46,269 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:46,270 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:46,270 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:46,270 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,270 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:46,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:46,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:46,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:46,273 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:46,273 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:46,273 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:46,273 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:46,273 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:46,273 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:46,274 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,274 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:46,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,274 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:46,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:46,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:46,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:46,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,275 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:46,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:46,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,276 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:46,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,283 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:46,283 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:46,283 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,283 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,283 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,283 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:46,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:46,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:46,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:46,283 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:46,283 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:46,283 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,284 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,284 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:46,284 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:46,284 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:46,284 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:46,362 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:46,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922546362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:46,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922546362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:46,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922546362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:46,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922546362, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,365 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:46,365 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922546365, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,384 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:46,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:46,386 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:46,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:46,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:46,387 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:46,387 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:46,387 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:46,387 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:46,388 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,388 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,389 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:46,389 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:46,389 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,389 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:46,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,390 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:46,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,390 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:46,390 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:46,390 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:46,390 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:46,390 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:46,390 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:46,390 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:46,391 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:46,391 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:46,391 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:46,391 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:46,391 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:46,391 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:46,391 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:46,391 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,391 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:46,391 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:46,393 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,393 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:46,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:46,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:46,393 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,393 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:46,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:46,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:46,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:46,395 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:46,395 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17be18e9[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:46,395 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,396 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,396 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:46,396 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:46,396 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,396 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:46,396 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:46,398 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:46,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:46,398 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:46,399 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:46,399 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:46,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:46,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:46,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:46,401 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:46,401 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:46,402 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:46,402 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:46,402 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:46,402 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:46,403 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,403 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,403 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,403 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:46,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:46,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:46,404 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:46,404 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:46,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:46,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,406 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:46,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,408 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,408 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,408 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:46,408 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,408 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:46,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:46,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:46,408 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:46,409 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:46,408 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:46,409 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:46,409 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,409 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,409 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,409 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:46,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:46,509 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:46,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:46,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:46,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:46,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:46,513 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:46,513 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:46,513 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:46,513 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:46,513 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,513 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,514 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,515 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:46,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:46,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:46,515 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,515 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:46,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:46,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:46,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:46,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:46,516 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:46,516 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:46,516 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:46,516 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:46,516 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:46,516 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:46,516 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:46,516 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:46,516 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:46,516 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:46,517 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:46,517 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,517 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:46,517 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:46,521 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,521 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:46,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:46,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:46,522 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,522 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:46,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:46,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:46,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:46,523 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@623e6852[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:46,523 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:46,523 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,524 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,524 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:46,524 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:46,524 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:46,524 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,524 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:46,526 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:46,526 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,526 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,526 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:46,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:46,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:46,528 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:46,528 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:46,528 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:46,528 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:46,528 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:46,528 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:46,530 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:46,530 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,530 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:46,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:46,530 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:46,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:46,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,532 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:46,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,535 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,535 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,535 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:46,535 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:46,535 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:46,535 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:46,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:46,535 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,535 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:46,535 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:46,535 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:46,536 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:46,536 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:46,536 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,536 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:46,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:46,628 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=13 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/23c0cb46ada34eb1a3790c429cecdd98 2023-04-08 02:54:46,636 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:46,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:46,637 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/21df7a61da0d40f88f0158b0ed3f0876 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/21df7a61da0d40f88f0158b0ed3f0876 2023-04-08 02:54:46,640 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:46,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:46,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:46,641 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:46,641 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:46,641 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:46,641 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:46,641 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,641 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,642 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/21df7a61da0d40f88f0158b0ed3f0876, entries=150, sequenceid=13, filesize=11.6 K 2023-04-08 02:54:46,642 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:46,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:46,643 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,643 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,643 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:46,643 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/ec2af5492feb49128a938cc0c35d349c as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/ec2af5492feb49128a938cc0c35d349c 2023-04-08 02:54:46,643 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:46,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:46,644 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:46,644 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:46,644 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:46,644 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:46,645 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:46,645 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:46,645 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:46,645 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:46,645 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:46,645 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:46,645 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:46,645 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:46,645 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,645 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:46,645 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:46,646 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,646 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:46,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:46,647 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,647 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:46,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:46,647 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/ec2af5492feb49128a938cc0c35d349c, entries=150, sequenceid=13, filesize=11.6 K 2023-04-08 02:54:46,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:46,648 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/23c0cb46ada34eb1a3790c429cecdd98 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/23c0cb46ada34eb1a3790c429cecdd98 2023-04-08 02:54:46,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:46,648 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:46,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@60eead6e[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:46,649 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:46,649 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,650 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,650 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:46,650 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:46,650 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:46,650 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,650 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:46,651 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/23c0cb46ada34eb1a3790c429cecdd98, entries=150, sequenceid=13, filesize=11.6 K 2023-04-08 02:54:46,651 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for d3e62a2534068dd21a6b490550cac850 in 925ms, sequenceid=13, compaction requested=false 2023-04-08 02:54:46,652 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:46,652 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,652 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:46,652 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:46,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:46,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:46,652 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:46,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:46,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:46,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:46,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:46,654 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:46,654 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:46,654 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:46,654 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:46,654 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:46,654 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:46,656 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,656 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,656 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:46,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:46,656 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:46,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,657 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:46,657 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:46,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:46,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:46,658 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:46,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,663 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:46,663 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:46,663 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:46,663 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:46,663 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,664 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,664 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,664 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:46,663 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:46,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:46,664 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,664 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:46,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:46,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:46,664 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,664 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:46,664 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:46,764 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:46,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:46,766 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:46,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:46,767 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:46,767 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:46,768 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:46,768 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:46,768 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:46,768 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,768 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,770 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:46,770 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:46,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:46,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,771 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,771 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:46,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:46,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,771 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:46,772 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:46,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:46,772 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:46,772 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:46,772 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:46,772 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:46,772 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:46,772 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:46,772 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:46,772 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-08 02:54:46,773 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:54:46,773 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:46,773 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:54:46,773 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:46,773 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:54:46,773 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:46,781 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/7b358e84c4794107b9dac44972fba574 2023-04-08 02:54:46,790 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/4a0acab50073490ebf42e95ac38e6c09 2023-04-08 02:54:46,802 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=33 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/17f0a683716249f98b90f6c2bfac2063 2023-04-08 02:54:46,817 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/7b358e84c4794107b9dac44972fba574 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/7b358e84c4794107b9dac44972fba574 2023-04-08 02:54:46,821 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/7b358e84c4794107b9dac44972fba574, entries=150, sequenceid=33, filesize=11.6 K 2023-04-08 02:54:46,826 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/4a0acab50073490ebf42e95ac38e6c09 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/4a0acab50073490ebf42e95ac38e6c09 2023-04-08 02:54:46,832 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/4a0acab50073490ebf42e95ac38e6c09, entries=150, sequenceid=33, filesize=11.6 K 2023-04-08 02:54:46,833 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/17f0a683716249f98b90f6c2bfac2063 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/17f0a683716249f98b90f6c2bfac2063 2023-04-08 02:54:46,837 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/17f0a683716249f98b90f6c2bfac2063, entries=150, sequenceid=33, filesize=11.6 K 2023-04-08 02:54:46,838 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for d3e62a2534068dd21a6b490550cac850 in 66ms, sequenceid=33, compaction requested=false 2023-04-08 02:54:46,838 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:46,838 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:46,839 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:46,839 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:46,839 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,839 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:46,839 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:46,841 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,841 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:46,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:46,842 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,842 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:46,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:46,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:46,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:46,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:46,843 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1f6aaed3[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:46,843 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:46,844 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,845 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,845 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,845 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:46,845 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:46,845 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:46,846 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,846 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:46,846 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:46,846 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:46,846 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:46,846 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:46,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:46,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:46,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:46,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:46,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:46,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:46,849 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:46,850 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:46,850 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:46,850 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:46,850 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:46,852 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,852 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,852 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:46,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:46,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:46,853 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:46,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:46,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:46,853 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:46,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,854 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:46,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,856 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:46,856 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:46,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:46,856 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,857 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,857 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:46,857 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:46,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:46,857 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:46,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:46,857 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,857 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,857 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,857 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,857 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,857 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:46,857 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:46,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:46,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:46,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:46,871 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-08 02:54:46,872 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:54:46,873 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:46,873 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:54:46,873 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:46,873 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:54:46,873 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:46,887 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:46,887 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:46,887 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:46,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922546885, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922546884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,887 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922546884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,889 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:46,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922546887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,889 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:46,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922546887, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,957 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:46,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:46,960 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:46,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:46,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:46,962 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:46,962 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:46,962 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:46,962 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:46,965 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,965 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,966 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:46,966 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:46,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:46,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,967 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,967 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:46,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:46,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:46,968 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:46,968 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:46,968 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:46,968 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:46,968 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:46,968 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:46,975 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:46,975 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:46,975 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:46,975 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:46,975 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:46,975 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:46,975 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:46,975 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:46,975 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,975 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:46,975 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:46,983 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:46,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:46,984 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,984 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,984 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:46,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:46,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:46,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:46,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:46,986 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@78f48487[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:46,986 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:46,986 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,987 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:46,988 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:46,988 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:46,988 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:46,988 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,988 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:46,989 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:46,989 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922546989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:46,990 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:46,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922546989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:46,989 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:46,990 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:46,990 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:46,990 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:46,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:46,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922546989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:46,992 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922546990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,992 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:46,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922546990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:46,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:46,993 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:46,993 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:46,993 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:46,993 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:46,993 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:46,995 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,995 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:46,995 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:46,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:46,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:46,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:46,996 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:46,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:46,997 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:46,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:46,999 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:46,999 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,000 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,000 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,000 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,000 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,000 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:47,000 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:47,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:47,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:47,000 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:47,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:47,000 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:47,000 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:47,000 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,000 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:47,000 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:47,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:47,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:47,100 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:47,101 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:47,103 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:47,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:47,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:47,105 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:47,105 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:47,105 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:47,105 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:47,106 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,106 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,107 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:47,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:47,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:47,108 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,108 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,108 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:47,109 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:47,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,109 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:47,110 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:47,110 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:47,110 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:47,110 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:47,110 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:47,113 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:47,113 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:47,113 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:47,113 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:47,113 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:47,113 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:47,113 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:47,113 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:47,113 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,113 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:47,113 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:47,115 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,115 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:47,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:47,115 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:47,115 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:47,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:47,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:47,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:47,117 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69991f33[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:47,117 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:47,117 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,118 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,118 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:47,118 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:47,118 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:47,118 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,118 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:47,120 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:47,120 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:47,120 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:47,120 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:47,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:47,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:47,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:47,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:47,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:47,124 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:47,124 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:47,124 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:47,124 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:47,124 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:47,124 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:47,126 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,126 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:47,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:47,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:47,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,127 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:47,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:47,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:47,128 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:47,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,130 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:47,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,133 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:47,133 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:47,133 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:47,133 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:47,133 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:47,133 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:47,133 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,133 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,133 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,133 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,133 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,133 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:47,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:47,133 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:47,134 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:47,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:47,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:47,191 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:47,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922547191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,192 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:47,192 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922547191, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,193 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:47,193 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922547192, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,193 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:47,194 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922547193, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,195 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:47,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922547194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,234 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:47,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:47,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:47,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:47,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:47,237 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:47,237 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:47,237 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:47,238 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:47,238 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,238 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,242 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:47,242 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:47,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:47,242 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,242 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:47,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:47,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,243 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:47,243 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:47,243 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:47,243 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:47,244 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:47,244 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:47,244 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:47,244 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:47,244 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:47,244 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:47,244 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:47,244 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:47,244 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:47,244 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:47,244 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,244 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:47,244 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:47,246 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,246 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:47,246 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,246 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:47,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:47,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:47,248 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c895ab3[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:47,248 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:47,248 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,251 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,251 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:47,251 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:47,251 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:47,252 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,252 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:47,252 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,252 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:47,252 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:47,252 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:47,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:47,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:47,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:47,256 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:47,256 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:47,256 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:47,256 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:47,256 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:47,257 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,257 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:47,257 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:47,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:47,258 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:47,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,259 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:47,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,262 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,262 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:47,262 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:47,262 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,263 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,263 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,263 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,263 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,263 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:47,262 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:47,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:47,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:47,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:47,263 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:47,263 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:47,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:47,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:47,282 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/93c0f59b6b774e66b2d7ca0cd15f6e92 2023-04-08 02:54:47,316 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/e834842d1dbd496a971e44156f3a3f75 2023-04-08 02:54:47,363 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:47,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:47,365 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:47,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:47,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:47,367 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:47,367 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:47,367 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:47,367 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:47,367 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,367 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,370 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:47,370 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:47,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:47,370 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,370 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:47,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:47,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:47,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:47,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:47,371 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:47,371 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:47,371 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:47,371 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:47,371 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:47,372 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:47,372 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:47,372 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:47,372 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:47,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:47,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:47,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:47,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:47,373 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,373 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:47,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:47,374 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,374 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:47,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:47,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:47,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:47,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:47,375 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:47,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36df7fc7[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:47,375 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,376 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,377 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:47,377 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:47,377 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:47,377 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,377 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:47,378 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,378 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:47,378 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:47,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:47,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:47,378 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:47,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:47,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:47,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:47,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:47,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:47,380 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:47,380 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:47,380 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:47,381 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:47,381 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:47,387 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,387 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,388 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:47,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:47,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:47,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:47,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:47,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:47,388 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:47,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,389 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,391 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,392 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,392 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,392 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:47,392 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:47,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:47,392 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:47,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:47,392 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:47,392 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,392 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,392 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,392 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:47,392 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:47,392 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:47,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:47,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:47,492 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:47,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:47,494 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:47,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922547492, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:47,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:47,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:47,496 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:47,496 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:47,496 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:47,496 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:47,496 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:47,496 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922547494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,496 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,496 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,496 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:47,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922547494, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:47,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922547497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,497 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:47,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922547497, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,498 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:47,498 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:47,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:47,498 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,498 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:47,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:47,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,499 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:47,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:47,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:47,504 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:47,504 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:47,504 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:47,516 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:47,516 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:47,516 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:47,516 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:47,516 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:47,516 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:47,516 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:47,516 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:47,516 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,516 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:47,516 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:47,519 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,519 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:47,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:47,520 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,520 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:47,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:47,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:47,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:47,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:47,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a531527[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:47,521 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:47,521 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,523 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,523 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:47,523 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:47,523 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:47,523 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,523 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:47,525 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,525 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:47,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:47,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:47,525 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:47,525 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:47,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:47,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:47,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:47,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:47,527 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:47,527 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:47,527 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:47,527 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:47,527 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:47,529 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,529 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,529 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:47,529 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:47,529 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:47,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:47,530 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,531 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,534 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,534 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:47,534 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:47,534 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:47,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:47,534 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:47,534 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,534 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,534 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,534 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,534 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,534 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:47,534 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:47,534 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:47,534 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:47,534 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:47,535 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:47,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:47,634 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:47,635 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:47,637 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:47,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:47,638 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:47,638 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:47,638 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:47,638 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:47,638 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:47,638 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,639 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,646 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:47,646 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:47,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:47,647 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,647 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:47,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,647 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:47,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:47,648 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:47,648 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:47,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:47,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:47,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:47,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:47,648 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:47,648 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:47,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:47,648 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:47,648 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:47,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:47,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:47,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:47,649 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:47,658 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,658 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:47,658 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,658 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:47,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,659 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:47,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:47,660 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b12ffd9[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:47,660 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:47,660 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,666 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,666 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,666 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,666 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:47,666 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:47,666 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:47,666 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:47,667 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,667 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:47,668 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:47,668 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:47,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:47,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:47,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:47,670 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:47,670 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:47,670 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:47,670 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:47,670 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:47,672 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,672 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:47,672 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:47,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:47,672 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:47,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,673 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:47,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,675 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,675 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:47,675 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:47,675 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,675 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:47,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:47,676 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:47,676 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:47,676 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,675 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:47,676 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:47,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:47,676 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:47,676 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,676 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,676 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:47,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:47,736 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=45 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/eafa802837c84e58acc988961514407d 2023-04-08 02:54:47,740 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/93c0f59b6b774e66b2d7ca0cd15f6e92 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/93c0f59b6b774e66b2d7ca0cd15f6e92 2023-04-08 02:54:47,744 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/93c0f59b6b774e66b2d7ca0cd15f6e92, entries=150, sequenceid=45, filesize=11.6 K 2023-04-08 02:54:47,745 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/e834842d1dbd496a971e44156f3a3f75 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e834842d1dbd496a971e44156f3a3f75 2023-04-08 02:54:47,748 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e834842d1dbd496a971e44156f3a3f75, entries=150, sequenceid=45, filesize=11.6 K 2023-04-08 02:54:47,749 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/eafa802837c84e58acc988961514407d as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/eafa802837c84e58acc988961514407d 2023-04-08 02:54:47,753 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/eafa802837c84e58acc988961514407d, entries=150, sequenceid=45, filesize=11.6 K 2023-04-08 02:54:47,754 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for d3e62a2534068dd21a6b490550cac850 in 883ms, sequenceid=45, compaction requested=true 2023-04-08 02:54:47,754 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:47,754 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:47,754 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:54:47,754 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:54:47,754 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:47,754 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:47,755 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:54:47,755 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/A is initiating minor compaction (all files) 2023-04-08 02:54:47,755 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:54:47,755 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/A in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:47,755 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/B is initiating minor compaction (all files) 2023-04-08 02:54:47,755 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/21df7a61da0d40f88f0158b0ed3f0876, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/7b358e84c4794107b9dac44972fba574, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/93c0f59b6b774e66b2d7ca0cd15f6e92] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=34.8 K 2023-04-08 02:54:47,755 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/B in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:47,755 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/ec2af5492feb49128a938cc0c35d349c, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/4a0acab50073490ebf42e95ac38e6c09, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e834842d1dbd496a971e44156f3a3f75] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=34.8 K 2023-04-08 02:54:47,756 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 21df7a61da0d40f88f0158b0ed3f0876, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680922485723 2023-04-08 02:54:47,756 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting ec2af5492feb49128a938cc0c35d349c, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680922485723 2023-04-08 02:54:47,756 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 7b358e84c4794107b9dac44972fba574, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680922485739 2023-04-08 02:54:47,756 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 4a0acab50073490ebf42e95ac38e6c09, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680922485739 2023-04-08 02:54:47,756 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 93c0f59b6b774e66b2d7ca0cd15f6e92, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1680922486868 2023-04-08 02:54:47,757 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting e834842d1dbd496a971e44156f3a3f75, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1680922486868 2023-04-08 02:54:47,764 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#A#compaction#218 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:47,765 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#B#compaction#219 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:47,776 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:47,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:47,777 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/f24dbbad89a846e8989762e099350978 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/f24dbbad89a846e8989762e099350978 2023-04-08 02:54:47,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:47,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:47,780 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:47,780 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:47,780 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:47,780 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:47,780 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:47,780 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,780 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,782 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/A of d3e62a2534068dd21a6b490550cac850 into f24dbbad89a846e8989762e099350978(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:47,782 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:47,783 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/A, priority=13, startTime=1680922487754; duration=0sec 2023-04-08 02:54:47,783 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:47,783 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:54:47,784 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35658 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:54:47,784 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/C is initiating minor compaction (all files) 2023-04-08 02:54:47,784 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/C in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:47,784 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/23c0cb46ada34eb1a3790c429cecdd98, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/17f0a683716249f98b90f6c2bfac2063, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/eafa802837c84e58acc988961514407d] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=34.8 K 2023-04-08 02:54:47,784 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,784 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 23c0cb46ada34eb1a3790c429cecdd98, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=13, earliestPutTs=1680922485723 2023-04-08 02:54:47,784 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:47,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:47,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:47,785 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 17f0a683716249f98b90f6c2bfac2063, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=33, earliestPutTs=1680922485739 2023-04-08 02:54:47,785 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,785 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:47,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,785 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting eafa802837c84e58acc988961514407d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1680922486868 2023-04-08 02:54:47,785 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:47,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,786 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:47,786 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:47,786 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:47,786 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:47,786 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:47,786 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:47,786 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:47,786 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:47,786 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:47,786 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:47,786 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-08 02:54:47,787 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:54:47,787 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:47,787 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:54:47,787 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:47,787 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:54:47,787 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:47,793 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#C#compaction#221 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:47,797 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/1f57ea055c0043b3b9b450ab6f386b40 2023-04-08 02:54:47,803 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/e8555f4db1b541c39bc6c752959b3dbb as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/e8555f4db1b541c39bc6c752959b3dbb 2023-04-08 02:54:47,810 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/C of d3e62a2534068dd21a6b490550cac850 into e8555f4db1b541c39bc6c752959b3dbb(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:47,810 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:47,810 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/C, priority=13, startTime=1680922487754; duration=0sec 2023-04-08 02:54:47,810 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:47,816 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/8feaee6405074216a582941bcc279d23 2023-04-08 02:54:47,830 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=66 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/2de979b47e894794a929205856c44c12 2023-04-08 02:54:47,834 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/1f57ea055c0043b3b9b450ab6f386b40 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/1f57ea055c0043b3b9b450ab6f386b40 2023-04-08 02:54:47,841 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/1f57ea055c0043b3b9b450ab6f386b40, entries=150, sequenceid=66, filesize=11.6 K 2023-04-08 02:54:47,842 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/8feaee6405074216a582941bcc279d23 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/8feaee6405074216a582941bcc279d23 2023-04-08 02:54:47,846 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/8feaee6405074216a582941bcc279d23, entries=150, sequenceid=66, filesize=11.6 K 2023-04-08 02:54:47,847 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/2de979b47e894794a929205856c44c12 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/2de979b47e894794a929205856c44c12 2023-04-08 02:54:47,850 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/2de979b47e894794a929205856c44c12, entries=150, sequenceid=66, filesize=11.6 K 2023-04-08 02:54:47,851 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=0 B/0 for d3e62a2534068dd21a6b490550cac850 in 65ms, sequenceid=66, compaction requested=false 2023-04-08 02:54:47,851 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:47,851 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:47,851 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:47,852 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:47,852 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,852 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:47,852 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:47,857 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,857 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:47,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:47,858 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,858 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:47,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:47,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:47,860 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d7868ca[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:47,860 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:47,861 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,862 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,862 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,862 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:47,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:47,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:47,863 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,863 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:47,864 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,864 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:47,864 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:47,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:47,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:47,864 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:47,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:47,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:47,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:47,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:47,868 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:47,868 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:47,868 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:47,868 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:47,868 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:47,869 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:47,876 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,876 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:47,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,876 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:47,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:47,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:47,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:47,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:47,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,878 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:47,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,879 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:47,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,882 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:47,882 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:47,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:47,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:47,883 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:47,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:47,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:47,883 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:47,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:47,883 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:47,883 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:47,883 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,883 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:47,883 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,883 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,883 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,883 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,983 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:47,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:47,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:47,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:47,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:47,987 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:47,987 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:47,987 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:47,987 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:47,987 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,987 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,988 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:47,988 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,989 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:47,989 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,989 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:47,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,989 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:47,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:47,990 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:47,990 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:47,990 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:47,990 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:47,990 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:47,990 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:47,990 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:47,990 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:47,990 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:47,990 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:47,991 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:47,991 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:47,991 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:47,991 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:47,991 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,991 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:47,991 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:47,992 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,992 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:47,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:47,992 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,993 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:47,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:47,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:47,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:47,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:47,994 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@309401e5[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:47,994 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:47,994 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,996 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:47,996 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:47,996 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:47,996 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:47,996 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,996 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:47,998 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,998 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:47,998 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:47,998 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:47,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:47,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:48,004 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-08 02:54:48,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:48,005 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:48,005 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:54:48,005 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:48,005 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:54:48,005 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:48,005 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:54:48,005 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:48,008 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:48,008 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:48,008 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:48,008 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:48,008 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:48,010 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,010 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:48,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:48,010 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,010 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:48,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,011 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:48,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:48,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:48,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:48,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,012 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:48,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,015 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:48,015 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:48,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:48,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:48,015 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,015 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:48,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:48,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:48,015 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:48,015 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:48,016 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:48,016 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,016 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:48,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:48,016 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,016 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,016 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,016 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,023 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922488016, max timestamp=1680922488022Num uniques 859; 2023-04-08 02:54:48,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922488016, max timestamp=1680922488022Num uniques 859; 2023-04-08 02:54:48,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922488016, max timestamp=1680922488022Num uniques 859; 2023-04-08 02:54:48,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:54:48,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:54:48,024 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:54:48,040 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:48,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922548036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,042 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:48,042 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:48,042 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922548039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922548039, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:48,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922548040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,043 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:48,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922548040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,116 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:48,117 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:48,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:48,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:48,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:48,119 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:48,119 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:48,119 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:48,120 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:48,120 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,120 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,122 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:48,122 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:48,122 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,122 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:48,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,123 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:48,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,123 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:48,123 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:48,124 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:48,124 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:48,124 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:48,124 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:48,124 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:48,124 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:48,124 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:48,124 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:48,124 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:48,124 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:48,124 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:48,124 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:48,124 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,124 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:48,124 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:48,126 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,126 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:48,126 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,126 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:48,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:48,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:48,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:48,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:48,128 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@429cc0fd[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:48,128 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:48,128 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,130 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,130 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:48,130 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,131 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:48,131 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:48,131 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:48,140 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,140 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:48,141 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:48,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:48,141 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:48,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:48,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,143 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:48,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922548141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:48,144 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:48,144 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:48,144 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:48,144 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:48,144 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:48,144 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:48,145 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,145 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:48,145 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:48,145 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922548143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,146 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,146 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:48,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:48,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:48,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922548144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,147 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:48,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:48,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:48,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922548144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922548144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:48,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:48,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,147 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,149 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:48,149 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,149 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,150 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,150 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,150 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,150 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,150 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:48,150 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:48,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:48,150 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:48,150 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:48,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:48,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:48,150 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:48,150 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:48,150 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:48,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:48,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:48,184 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/e4ddbf02c5454155a0a7f145ff80683d as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e4ddbf02c5454155a0a7f145ff80683d 2023-04-08 02:54:48,188 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/B of d3e62a2534068dd21a6b490550cac850 into e4ddbf02c5454155a0a7f145ff80683d(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:48,188 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:48,188 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/B, priority=13, startTime=1680922487754; duration=0sec 2023-04-08 02:54:48,189 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:48,251 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:48,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:48,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:48,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:48,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:48,254 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:48,254 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:48,254 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:48,254 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:48,254 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,254 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,256 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:48,256 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:48,256 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,256 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:48,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:48,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,257 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:48,257 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:48,257 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:48,257 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:48,257 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:48,257 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:48,257 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:48,257 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:48,257 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:48,257 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:48,257 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:48,257 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:48,257 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:48,258 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:48,258 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,258 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:48,258 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:48,260 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,260 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:48,260 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,260 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:48,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:48,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:48,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:48,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eac1340[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:48,261 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:48,262 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,262 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,263 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:48,263 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:48,263 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:48,263 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,263 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:48,264 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,264 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:48,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:48,264 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:48,264 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:48,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:48,266 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:48,266 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:48,267 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:48,267 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:48,267 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:48,267 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:48,268 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,268 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:48,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:48,268 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:48,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:48,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:48,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:48,269 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:48,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,270 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:48,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,273 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:48,273 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,273 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,273 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:48,273 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,273 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:48,273 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:48,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:48,273 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:48,273 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:48,273 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,273 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:48,273 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,274 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,274 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:48,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:48,274 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:48,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:48,345 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:48,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922548345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:48,349 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:48,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922548347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922548347, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,349 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:48,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922548348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,350 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:48,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922548349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,374 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:48,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:48,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:48,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:48,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:48,377 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:48,377 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:48,377 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:48,377 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:48,378 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,378 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,379 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:48,379 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:48,379 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,379 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:48,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:48,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,380 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:48,380 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:48,380 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:48,380 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:48,380 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:48,380 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:48,381 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:48,381 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:48,381 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:48,381 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:48,381 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:48,381 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:48,381 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:48,381 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:48,381 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,381 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:48,381 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:48,382 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,382 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:48,383 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,383 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:48,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:48,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:48,384 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:48,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@202c0667[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:48,384 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,385 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,386 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:48,386 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:48,386 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,386 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:48,386 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:48,388 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,388 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:48,388 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:48,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:48,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:48,388 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:48,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:48,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:48,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:48,390 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:48,390 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:48,390 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:48,390 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:48,391 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:48,391 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:48,392 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,392 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:48,392 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:48,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:48,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:48,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:48,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:48,393 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:48,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:48,395 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,397 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:48,397 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:48,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:48,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:48,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:48,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:48,397 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,397 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,397 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:48,397 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:48,397 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:48,397 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:48,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:48,397 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,397 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,397 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,397 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,426 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/3659dde192c243f19be7a4301bfba1a7 2023-04-08 02:54:48,497 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:48,498 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:48,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:48,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:48,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:48,500 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:48,500 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:48,500 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:48,501 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:48,501 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,501 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,502 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,502 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:48,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:48,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:48,503 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,503 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:48,503 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:48,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,503 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:48,504 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:48,504 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:48,504 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:48,504 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:48,504 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:48,504 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:48,504 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:48,504 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:48,504 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:48,504 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:48,504 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:48,504 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:48,504 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:48,504 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,504 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:48,504 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:48,507 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,507 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:48,507 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,507 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:48,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:48,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:48,508 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:48,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30a7b0ae[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:48,508 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,509 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,509 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:48,509 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:48,510 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:48,510 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,510 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:48,511 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,511 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:48,511 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:48,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,511 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:48,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:48,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:48,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:48,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:48,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:48,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:48,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:48,513 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:48,513 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:48,513 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:48,513 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:48,513 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:48,515 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,515 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:48,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,515 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:48,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:48,515 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:48,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:48,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,516 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:48,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,522 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,522 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:48,522 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,522 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:48,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:48,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:48,522 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,522 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,522 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,522 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,522 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:48,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:48,522 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:48,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:48,522 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:48,522 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:48,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:48,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:48,623 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:48,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:48,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:48,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:48,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:48,626 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:48,626 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:48,626 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:48,626 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:48,626 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,626 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,628 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:48,628 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:48,628 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,628 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:48,628 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:48,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,629 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:48,629 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:48,629 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:48,629 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:48,629 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:48,629 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:48,629 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:48,629 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:48,629 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:48,629 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:48,629 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:48,630 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:48,630 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:48,630 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:48,630 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,630 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:48,630 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:48,632 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,632 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:48,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:48,632 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,632 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:48,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:48,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:48,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:48,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:48,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1498056f[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:48,634 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:48,634 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,635 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,635 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:48,635 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:48,635 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:48,635 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,635 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:48,637 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:48,637 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:48,637 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:48,640 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:48,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:48,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:48,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:48,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:48,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:48,642 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:48,642 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:48,642 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:48,642 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:48,642 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:48,642 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:48,644 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,644 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,644 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:48,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:48,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:48,644 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:48,646 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,648 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,648 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,648 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,648 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,648 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:48,648 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:48,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:48,648 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,648 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:48,649 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:48,649 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:48,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:48,649 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:48,649 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:48,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:48,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:48,649 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:48,649 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922548647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:48,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922548650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:48,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922548650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:48,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922548650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,654 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:48,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922548652, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,749 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:48,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:48,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:48,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:48,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:48,752 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:48,752 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:48,752 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:48,752 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:48,752 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,752 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,754 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:48,754 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:48,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:48,754 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,754 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:48,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:48,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,755 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:48,755 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:48,755 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:48,755 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:48,755 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:48,755 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:48,755 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:48,755 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:48,755 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:48,755 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:48,755 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:48,756 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:48,756 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:48,756 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:48,756 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,756 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:48,756 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:48,758 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,758 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:48,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:48,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:48,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:48,760 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a5032af[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:48,760 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:48,760 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,761 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:48,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:48,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:48,761 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,761 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:48,763 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:48,763 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:48,763 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:48,763 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:48,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:48,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:48,765 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:48,765 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:48,765 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:48,765 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:48,765 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:48,766 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,766 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:48,766 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:48,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:48,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:48,767 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,768 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:48,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,771 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,771 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:48,771 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,771 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:48,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:48,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:48,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:48,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:48,771 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:48,771 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:48,771 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,771 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,771 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:48,771 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:48,771 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,771 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:48,772 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:48,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:48,836 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/db8f8a73d85f4cd1b5fe235697d24d58 2023-04-08 02:54:48,847 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/7cb603f1b1704b5cacf1955aee6a6426 2023-04-08 02:54:48,851 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/3659dde192c243f19be7a4301bfba1a7 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/3659dde192c243f19be7a4301bfba1a7 2023-04-08 02:54:48,854 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/3659dde192c243f19be7a4301bfba1a7, entries=200, sequenceid=79, filesize=13.9 K 2023-04-08 02:54:48,855 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/db8f8a73d85f4cd1b5fe235697d24d58 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/db8f8a73d85f4cd1b5fe235697d24d58 2023-04-08 02:54:48,859 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/db8f8a73d85f4cd1b5fe235697d24d58, entries=150, sequenceid=79, filesize=11.6 K 2023-04-08 02:54:48,860 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/7cb603f1b1704b5cacf1955aee6a6426 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/7cb603f1b1704b5cacf1955aee6a6426 2023-04-08 02:54:48,863 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/7cb603f1b1704b5cacf1955aee6a6426, entries=150, sequenceid=79, filesize=11.6 K 2023-04-08 02:54:48,864 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for d3e62a2534068dd21a6b490550cac850 in 860ms, sequenceid=79, compaction requested=true 2023-04-08 02:54:48,864 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:48,864 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:48,864 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:54:48,864 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:54:48,864 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:48,864 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:48,865 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38101 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:54:48,865 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/A is initiating minor compaction (all files) 2023-04-08 02:54:48,865 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:54:48,865 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/A in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:48,865 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/B is initiating minor compaction (all files) 2023-04-08 02:54:48,865 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/f24dbbad89a846e8989762e099350978, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/1f57ea055c0043b3b9b450ab6f386b40, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/3659dde192c243f19be7a4301bfba1a7] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=37.2 K 2023-04-08 02:54:48,865 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/B in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:48,865 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e4ddbf02c5454155a0a7f145ff80683d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/8feaee6405074216a582941bcc279d23, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/db8f8a73d85f4cd1b5fe235697d24d58] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=34.9 K 2023-04-08 02:54:48,865 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting f24dbbad89a846e8989762e099350978, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1680922486868 2023-04-08 02:54:48,866 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting e4ddbf02c5454155a0a7f145ff80683d, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1680922486868 2023-04-08 02:54:48,866 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 1f57ea055c0043b3b9b450ab6f386b40, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680922486882 2023-04-08 02:54:48,866 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 3659dde192c243f19be7a4301bfba1a7, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1680922488000 2023-04-08 02:54:48,866 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 8feaee6405074216a582941bcc279d23, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680922486882 2023-04-08 02:54:48,866 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting db8f8a73d85f4cd1b5fe235697d24d58, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1680922488002 2023-04-08 02:54:48,872 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:48,872 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#A#compaction#227 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:48,872 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#B#compaction#228 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:48,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:48,874 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:48,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:48,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:48,880 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:48,880 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:48,880 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:48,880 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:48,880 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,881 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,882 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:48,882 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:48,882 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:48,882 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:48,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,883 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:48,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:48,883 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:48,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:48,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:48,883 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:48,883 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:48,884 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:48,887 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:48,887 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:48,887 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:48,888 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:48,888 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-08 02:54:48,888 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:54:48,888 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:54:48,888 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:54:48,888 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:54:48,888 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:54:48,889 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:54:48,892 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/da1228e4bb1f4aa48345db8b302ffa3f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/da1228e4bb1f4aa48345db8b302ffa3f 2023-04-08 02:54:48,896 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/B of d3e62a2534068dd21a6b490550cac850 into da1228e4bb1f4aa48345db8b302ffa3f(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:48,896 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:48,896 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/B, priority=13, startTime=1680922488864; duration=0sec 2023-04-08 02:54:48,896 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:48,896 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:54:48,897 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35761 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:54:48,897 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/C is initiating minor compaction (all files) 2023-04-08 02:54:48,897 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/C in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:48,897 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/e8555f4db1b541c39bc6c752959b3dbb, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/2de979b47e894794a929205856c44c12, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/7cb603f1b1704b5cacf1955aee6a6426] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=34.9 K 2023-04-08 02:54:48,897 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting e8555f4db1b541c39bc6c752959b3dbb, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=45, earliestPutTs=1680922486868 2023-04-08 02:54:48,898 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 2de979b47e894794a929205856c44c12, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=66, earliestPutTs=1680922486882 2023-04-08 02:54:48,898 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 7cb603f1b1704b5cacf1955aee6a6426, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1680922488002 2023-04-08 02:54:48,917 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#C#compaction#230 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:48,935 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/29efa88951b74ba18a0de24371408fa0 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/29efa88951b74ba18a0de24371408fa0 2023-04-08 02:54:48,941 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/C of d3e62a2534068dd21a6b490550cac850 into 29efa88951b74ba18a0de24371408fa0(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:48,942 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:48,942 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/C, priority=13, startTime=1680922488864; duration=0sec 2023-04-08 02:54:48,942 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:49,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:49,154 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:49,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:49,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922549159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,164 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:49,164 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:49,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922549161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922549161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:49,165 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:49,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922549161, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922549162, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,265 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:49,265 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922549263, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,267 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:49,267 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922549266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,267 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:49,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922549266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:49,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922549266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,268 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:49,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922549266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,284 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/aee3fd6cbad14eceb604d3495bfc2f87 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/aee3fd6cbad14eceb604d3495bfc2f87 2023-04-08 02:54:49,289 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/A of d3e62a2534068dd21a6b490550cac850 into aee3fd6cbad14eceb604d3495bfc2f87(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:49,289 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:49,289 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/A, priority=13, startTime=1680922488864; duration=0sec 2023-04-08 02:54:49,289 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:49,314 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/2dfed944a4174da8834bc2c5c3150785 2023-04-08 02:54:49,467 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:49,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922549467, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,470 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:49,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922549468, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:49,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922549469, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,470 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:49,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922549470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,471 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:49,471 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922549470, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,551 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-08 02:54:49,742 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/d9e23cab6e4c461a9db0439057b0d869 2023-04-08 02:54:49,754 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=117 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/417c08dd7e434b26aa3e37b3d2b423ce 2023-04-08 02:54:49,759 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/2dfed944a4174da8834bc2c5c3150785 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/2dfed944a4174da8834bc2c5c3150785 2023-04-08 02:54:49,763 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/2dfed944a4174da8834bc2c5c3150785, entries=150, sequenceid=117, filesize=11.6 K 2023-04-08 02:54:49,763 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/d9e23cab6e4c461a9db0439057b0d869 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d9e23cab6e4c461a9db0439057b0d869 2023-04-08 02:54:49,768 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d9e23cab6e4c461a9db0439057b0d869, entries=150, sequenceid=117, filesize=11.6 K 2023-04-08 02:54:49,770 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/417c08dd7e434b26aa3e37b3d2b423ce as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/417c08dd7e434b26aa3e37b3d2b423ce 2023-04-08 02:54:49,771 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:49,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922549769, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:49,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922549772, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:49,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922549773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,773 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:49,773 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922549773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,774 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:49,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922549774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,775 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/417c08dd7e434b26aa3e37b3d2b423ce, entries=150, sequenceid=117, filesize=11.6 K 2023-04-08 02:54:49,776 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for d3e62a2534068dd21a6b490550cac850 in 888ms, sequenceid=117, compaction requested=false 2023-04-08 02:54:49,776 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:49,776 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:49,776 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:49,776 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:49,776 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,776 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:49,776 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:49,777 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:49,778 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:49,778 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:49,778 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:49,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:49,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:49,779 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2a0f1ebc[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:49,779 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:49,779 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:49,781 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:49,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:49,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:49,781 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:49,782 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:49,782 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,782 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:49,782 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:49,783 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:49,783 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:49,783 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,783 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:49,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:49,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:49,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:49,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:49,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:49,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:49,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:49,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:49,786 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:49,786 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:49,786 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:49,786 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:49,786 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:49,786 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:49,788 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:49,788 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:49,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:49,788 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:49,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:49,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:49,789 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:49,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:49,790 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:49,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:49,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,797 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:49,797 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:49,797 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,797 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:49,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:49,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:49,797 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:49,797 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:49,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:49,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:49,797 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,798 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:49,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:49,798 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:49,798 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:49,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:49,798 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:49,798 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:49,798 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:49,898 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:49,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:49,901 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:49,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:49,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:49,903 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:49,903 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:49,903 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:49,903 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:49,904 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:49,904 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:49,905 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,905 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:49,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:49,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:49,906 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:49,906 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:49,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:49,906 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:49,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:49,907 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:49,908 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:49,908 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:49,908 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:49,908 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:49,908 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:49,908 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:49,908 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:49,909 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:49,909 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:49,909 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-08 02:54:49,910 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:54:49,910 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:49,910 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:54:49,910 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:49,910 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:54:49,910 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:49,923 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/423319047a3040c68e582e118623c814 2023-04-08 02:54:50,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:50,276 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:50,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:50,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922550287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922490285, max timestamp=1680922490289Num uniques 859; 2023-04-08 02:54:50,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922490285, max timestamp=1680922490289Num uniques 859; 2023-04-08 02:54:50,290 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:50,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922490285, max timestamp=1680922490289Num uniques 859; 2023-04-08 02:54:50,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:54:50,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:54:50,290 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:54:50,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922550287, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,311 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:50,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922550307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:50,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922550309, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,312 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:50,312 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922550310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,335 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/b80eaaed15b04bc4bc5946ddae8f2265 2023-04-08 02:54:50,392 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:50,394 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922550392, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,395 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:50,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922550394, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,413 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:50,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922550412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,413 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:50,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922550412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,414 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:50,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922550413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,598 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:50,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922550596, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,598 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:50,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922550597, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,616 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:50,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922550614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:50,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922550615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,617 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:50,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922550615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,746 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/7040a0f67a0d414e845399f126ad5c31 2023-04-08 02:54:50,751 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/423319047a3040c68e582e118623c814 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/423319047a3040c68e582e118623c814 2023-04-08 02:54:50,756 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/423319047a3040c68e582e118623c814, entries=150, sequenceid=130, filesize=11.6 K 2023-04-08 02:54:50,757 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/b80eaaed15b04bc4bc5946ddae8f2265 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/b80eaaed15b04bc4bc5946ddae8f2265 2023-04-08 02:54:50,761 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/b80eaaed15b04bc4bc5946ddae8f2265, entries=150, sequenceid=130, filesize=11.6 K 2023-04-08 02:54:50,762 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/7040a0f67a0d414e845399f126ad5c31 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/7040a0f67a0d414e845399f126ad5c31 2023-04-08 02:54:50,766 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/7040a0f67a0d414e845399f126ad5c31, entries=150, sequenceid=130, filesize=11.6 K 2023-04-08 02:54:50,767 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for d3e62a2534068dd21a6b490550cac850 in 858ms, sequenceid=130, compaction requested=true 2023-04-08 02:54:50,767 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:50,767 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:50,767 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:50,767 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:50,767 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,767 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:50,767 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:50,769 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,769 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:50,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:50,769 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:50,770 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:50,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:50,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:50,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:50,772 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2720f993[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:50,772 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:50,772 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:50,773 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:50,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:50,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:50,773 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:50,773 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:50,773 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:50,774 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,774 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:50,776 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:50,776 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:50,776 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:50,776 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:50,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:50,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:50,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:50,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:50,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:50,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:50,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:50,785 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:50,785 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:50,786 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:50,786 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:50,786 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:50,786 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:50,788 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:50,788 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:50,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:50,788 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:50,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:50,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:50,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:50,789 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:50,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:50,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:50,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:50,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,791 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:50,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:50,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,794 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:50,794 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:50,794 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:50,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:50,794 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:50,794 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:50,794 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:50,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:50,794 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:50,794 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:50,794 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:50,794 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:50,794 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,794 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:50,794 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:50,794 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:50,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:50,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:50,894 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:50,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:50,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:50,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:50,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:50,898 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:50,898 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:50,898 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:50,898 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:50,898 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:50,898 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:50,900 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,900 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:50,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:50,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:50,900 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,900 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:50,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:50,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:50,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:50,902 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:50,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:50,902 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:50,902 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-08 02:54:50,902 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:50,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:50,902 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:54:50,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:50,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:50,902 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-08 02:54:50,903 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:54:50,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680922490296, max timestamp=1680922490901Num uniques 1018; 2023-04-08 02:54:50,903 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=3, new segment=null 2023-04-08 02:54:50,903 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:50,903 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:50,903 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:50,904 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:50,903 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:50,904 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:50,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-08 02:54:50,903 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:54:50,904 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:50,904 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:50,904 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:54:50,904 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,904 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:50,904 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:50,907 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:50,907 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:50,907 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:50,907 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:50,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:50,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:50,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:50,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:50,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:50,910 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73c73365[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:50,910 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:50,910 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:50,911 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:50,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:50,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:50,911 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:50,911 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:50,911 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:50,911 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,911 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:50,913 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/7a256b999e9540a581c63623c8099904 2023-04-08 02:54:50,914 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,914 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:50,914 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:50,914 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:50,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:50,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:50,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:50,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:50,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:50,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:50,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:50,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:50,919 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:50,919 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:50,919 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:50,919 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:50,919 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:50,919 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:50,921 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:50,921 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:50,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:50,921 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:50,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:50,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:50,922 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:50,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:50,923 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,924 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922550919, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:50,924 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:50,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:50,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922550922, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,926 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:50,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922550923, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,926 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:50,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922550924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,927 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:50,927 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922550924, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,927 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:50,928 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,928 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:50,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:50,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:50,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:50,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:50,928 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:50,928 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:50,928 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:50,928 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:50,928 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:50,928 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:50,928 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:50,928 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:50,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:50,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:50,928 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:50,930 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=46.96 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/2538b16666604a7a9b9b5ead38b6acec 2023-04-08 02:54:50,942 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=168 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/1a5377a20f294ac988dd6c5f60beccf0 2023-04-08 02:54:50,948 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/7a256b999e9540a581c63623c8099904 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/7a256b999e9540a581c63623c8099904 2023-04-08 02:54:50,955 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/7a256b999e9540a581c63623c8099904, entries=150, sequenceid=168, filesize=11.8 K 2023-04-08 02:54:50,956 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/2538b16666604a7a9b9b5ead38b6acec as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/2538b16666604a7a9b9b5ead38b6acec 2023-04-08 02:54:50,962 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/2538b16666604a7a9b9b5ead38b6acec, entries=150, sequenceid=168, filesize=11.8 K 2023-04-08 02:54:50,964 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/1a5377a20f294ac988dd6c5f60beccf0 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/1a5377a20f294ac988dd6c5f60beccf0 2023-04-08 02:54:50,972 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/1a5377a20f294ac988dd6c5f60beccf0, entries=150, sequenceid=168, filesize=11.8 K 2023-04-08 02:54:50,973 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~102.87 KB/105340, heapSize ~292.16 KB/299168, currentSize=80.51 KB/82440 for d3e62a2534068dd21a6b490550cac850 in 71ms, sequenceid=168, compaction requested=true 2023-04-08 02:54:50,973 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:50,973 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:50,973 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:54:50,973 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:54:50,973 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:50,973 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:50,974 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:54:50,974 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/B is initiating minor compaction (all files) 2023-04-08 02:54:50,974 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:54:50,974 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/B in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:50,974 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/A is initiating minor compaction (all files) 2023-04-08 02:54:50,974 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/da1228e4bb1f4aa48345db8b302ffa3f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d9e23cab6e4c461a9db0439057b0d869, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/b80eaaed15b04bc4bc5946ddae8f2265, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/2538b16666604a7a9b9b5ead38b6acec] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=46.8 K 2023-04-08 02:54:50,975 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/A in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:50,975 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/aee3fd6cbad14eceb604d3495bfc2f87, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/2dfed944a4174da8834bc2c5c3150785, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/423319047a3040c68e582e118623c814, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/7a256b999e9540a581c63623c8099904] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=46.8 K 2023-04-08 02:54:50,975 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting da1228e4bb1f4aa48345db8b302ffa3f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1680922488002 2023-04-08 02:54:50,975 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting aee3fd6cbad14eceb604d3495bfc2f87, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1680922488002 2023-04-08 02:54:50,975 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting d9e23cab6e4c461a9db0439057b0d869, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1680922488038 2023-04-08 02:54:50,975 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 2dfed944a4174da8834bc2c5c3150785, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1680922488038 2023-04-08 02:54:50,976 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting b80eaaed15b04bc4bc5946ddae8f2265, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680922489153 2023-04-08 02:54:50,976 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 423319047a3040c68e582e118623c814, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680922489153 2023-04-08 02:54:50,976 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 2538b16666604a7a9b9b5ead38b6acec, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680922490296 2023-04-08 02:54:50,976 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 7a256b999e9540a581c63623c8099904, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680922490296 2023-04-08 02:54:50,993 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#B#compaction#239 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:50,995 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#A#compaction#240 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:51,026 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/51ba9f71b0eb4b71b7fad023bdc60caa as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/51ba9f71b0eb4b71b7fad023bdc60caa 2023-04-08 02:54:51,029 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:51,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:51,029 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-08 02:54:51,029 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:54:51,029 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:51,029 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:54:51,029 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:51,029 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:54:51,029 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:51,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:51,032 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d3e62a2534068dd21a6b490550cac850/B of d3e62a2534068dd21a6b490550cac850 into 51ba9f71b0eb4b71b7fad023bdc60caa(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:51,032 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:51,032 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/B, priority=12, startTime=1680922490973; duration=0sec 2023-04-08 02:54:51,032 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:51,033 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:54:51,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:51,034 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47900 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:54:51,034 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/C is initiating minor compaction (all files) 2023-04-08 02:54:51,034 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/C in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:51,034 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/29efa88951b74ba18a0de24371408fa0, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/417c08dd7e434b26aa3e37b3d2b423ce, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/7040a0f67a0d414e845399f126ad5c31, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/1a5377a20f294ac988dd6c5f60beccf0] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=46.8 K 2023-04-08 02:54:51,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:51,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:51,035 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:51,035 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:51,035 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 29efa88951b74ba18a0de24371408fa0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1680922488002 2023-04-08 02:54:51,035 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:51,035 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:51,035 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 417c08dd7e434b26aa3e37b3d2b423ce, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=117, earliestPutTs=1680922488038 2023-04-08 02:54:51,036 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 7040a0f67a0d414e845399f126ad5c31, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680922489153 2023-04-08 02:54:51,036 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 1a5377a20f294ac988dd6c5f60beccf0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680922490296 2023-04-08 02:54:51,037 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,037 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:51,039 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,039 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:51,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:51,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:51,039 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,039 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:51,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:51,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:51,040 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:51,041 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/ca1cc1c2640545e29c36a791ab2f91c8 2023-04-08 02:54:51,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:51,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:51,041 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:51,041 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:51,041 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:51,041 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:51,041 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:51,042 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:51,042 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:51,042 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:51,042 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:51,042 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:51,042 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:51,042 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,042 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:51,042 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:51,044 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,044 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:51,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:51,045 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,045 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:51,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:51,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:51,045 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#C#compaction#242 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:51,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:51,047 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18e9b2f8[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:51,047 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:51,047 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,048 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,048 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,048 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,048 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:51,048 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:51,048 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:51,049 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,049 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:51,050 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:51,050 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:51,050 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:51,050 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:51,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:51,051 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:51,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922551047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:51,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:51,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:51,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922551050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:51,052 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:51,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922551050, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:51,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:51,053 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:51,053 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:51,053 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:51,053 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:51,053 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:51,053 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:51,054 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:51,054 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922551051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,055 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,055 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:51,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:51,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,056 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:51,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:51,056 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:51,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:51,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:51,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,059 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:51,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:51,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,062 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/f54b65712bda43bdb983609884a3a536 2023-04-08 02:54:51,062 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,062 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:51,062 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:51,062 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,062 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,062 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:51,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:51,062 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:51,062 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:51,063 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:51,062 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:51,062 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:51,063 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:51,063 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:51,063 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:51,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:51,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:51,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922551153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:51,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922551153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,155 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:51,155 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922551153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,157 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:51,158 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922551155, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,163 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:51,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:51,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:51,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:51,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:51,167 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:51,167 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:51,167 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:51,167 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:51,168 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,168 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:51,171 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,171 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:51,171 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:51,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:51,172 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,172 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:51,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:51,172 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:51,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:51,172 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:51,173 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:51,173 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:51,173 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:51,173 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:51,173 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:51,174 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:51,174 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:51,174 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:51,174 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:51,174 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:51,174 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:51,174 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:51,174 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:51,174 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,174 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:51,174 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:51,179 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,179 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:51,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:51,179 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,179 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:51,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:51,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:51,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a41a5a7[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:51,181 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:51,181 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,182 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,182 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:51,182 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,182 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:51,182 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:51,182 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:51,184 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:51,184 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,184 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:51,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:51,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:51,184 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:51,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:51,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:51,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:51,186 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:51,186 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:51,186 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:51,186 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:51,186 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:51,186 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:51,188 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,188 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,188 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:51,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:51,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:51,188 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:51,189 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:51,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,192 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,192 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:51,192 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:51,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:51,192 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:51,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:51,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:51,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:51,192 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:51,192 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:51,192 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,192 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,192 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,192 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:51,193 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:51,192 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:51,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:51,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:51,293 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:51,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:51,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:51,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:51,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:51,297 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:51,297 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:51,297 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:51,297 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:51,297 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,297 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:51,298 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:51,298 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:51,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:51,299 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,299 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:51,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:51,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:51,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:51,300 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:51,300 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:51,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:51,300 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:51,300 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:51,300 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:51,300 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:51,301 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:51,301 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:51,301 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:51,301 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:51,301 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:51,301 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:51,301 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:51,301 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,301 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:51,301 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:51,302 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,303 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:51,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:51,303 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,303 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:51,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:51,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:51,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:51,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:51,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:51,306 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a8d311d[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:51,306 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:51,306 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,307 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,307 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:51,307 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:51,307 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:51,307 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,307 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:51,309 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,309 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:51,309 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:51,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:51,309 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:51,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:51,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:51,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:51,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:51,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:51,312 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:51,312 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:51,312 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:51,312 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:51,312 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:51,314 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,314 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,314 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:51,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:51,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:51,315 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:51,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:51,316 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:51,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,319 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,319 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:51,319 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:51,319 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:51,319 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:51,319 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:51,319 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:51,319 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:51,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:51,319 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:51,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:51,319 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,319 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,319 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,319 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:51,320 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:51,320 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:51,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:51,358 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:51,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922551357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,359 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:51,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922551357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,359 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:51,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922551357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,362 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:51,362 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922551360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,420 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:51,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:51,422 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:51,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:51,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:51,424 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:51,424 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:51,424 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:51,424 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:51,424 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,425 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:51,427 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,427 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:51,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:51,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:51,427 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,427 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:51,428 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/e2418d2df4d44f5f86aa0805c657e8b6 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/e2418d2df4d44f5f86aa0805c657e8b6 2023-04-08 02:54:51,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:51,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:51,428 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:51,428 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:51,428 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:51,429 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:51,429 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:51,429 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:51,429 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:51,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922551427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,429 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:51,429 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:51,429 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:51,429 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:51,429 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:51,430 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:51,430 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:51,430 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:51,430 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,430 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:51,430 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:51,431 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,431 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:51,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:51,431 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,431 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:51,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:51,433 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d3e62a2534068dd21a6b490550cac850/A of d3e62a2534068dd21a6b490550cac850 into e2418d2df4d44f5f86aa0805c657e8b6(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:51,433 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:51,433 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/A, priority=12, startTime=1680922490973; duration=0sec 2023-04-08 02:54:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:51,433 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:51,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:51,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:51,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@209b6b32[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:51,434 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:51,434 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,435 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,435 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:51,435 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:51,435 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:51,436 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,436 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:51,437 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,437 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:51,437 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:51,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:51,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:51,437 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:51,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:51,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:51,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:51,440 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:51,440 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:51,440 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:51,440 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:51,440 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:51,440 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:51,443 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,443 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:51,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:51,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,444 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:51,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:51,445 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:51,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:51,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:51,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,446 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:51,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:51,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,450 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:51,450 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:51,450 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:51,450 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:51,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:51,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:51,450 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:51,450 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:51,450 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:51,450 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:51,450 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:51,450 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,450 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,450 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:51,450 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:51,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:51,459 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/48379221b677477bb49c122e4a792eb2 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/48379221b677477bb49c122e4a792eb2 2023-04-08 02:54:51,474 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=184 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/b9d6edbc704c4e66b7abc835d4a06502 2023-04-08 02:54:51,479 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d3e62a2534068dd21a6b490550cac850/C of d3e62a2534068dd21a6b490550cac850 into 48379221b677477bb49c122e4a792eb2(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:51,480 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:51,480 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/C, priority=12, startTime=1680922490973; duration=0sec 2023-04-08 02:54:51,480 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:51,482 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/ca1cc1c2640545e29c36a791ab2f91c8 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/ca1cc1c2640545e29c36a791ab2f91c8 2023-04-08 02:54:51,488 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/ca1cc1c2640545e29c36a791ab2f91c8, entries=150, sequenceid=184, filesize=11.8 K 2023-04-08 02:54:51,489 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/f54b65712bda43bdb983609884a3a536 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/f54b65712bda43bdb983609884a3a536 2023-04-08 02:54:51,498 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/f54b65712bda43bdb983609884a3a536, entries=150, sequenceid=184, filesize=11.8 K 2023-04-08 02:54:51,499 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/b9d6edbc704c4e66b7abc835d4a06502 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/b9d6edbc704c4e66b7abc835d4a06502 2023-04-08 02:54:51,506 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/b9d6edbc704c4e66b7abc835d4a06502, entries=150, sequenceid=184, filesize=11.8 K 2023-04-08 02:54:51,512 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for d3e62a2534068dd21a6b490550cac850 in 483ms, sequenceid=184, compaction requested=false 2023-04-08 02:54:51,512 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:51,550 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:51,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:51,553 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:51,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:51,554 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:51,554 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:51,554 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:51,554 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:51,554 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:51,555 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,555 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:51,556 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:51,556 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:51,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:51,556 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,556 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:51,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:51,557 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:51,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:51,557 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:51,557 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:51,557 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:51,558 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:51,558 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:51,558 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:51,558 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:51,558 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:51,558 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:51,558 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:51,558 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-08 02:54:51,558 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:54:51,559 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:51,559 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:54:51,559 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:51,559 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:54:51,559 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:51,570 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/31774130daf2498b93908415de5fc15f 2023-04-08 02:54:51,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:51,661 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:51,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:51,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922551671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:51,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922551671, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,675 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:51,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922551672, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,676 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:51,676 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922551673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:51,777 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922551775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,777 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:51,778 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922551775, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,778 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:51,779 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922551776, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,783 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:51,783 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922551782, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:51,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922551978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,980 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:51,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922551978, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:51,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922551979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,982 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/95cb51ce57f24bbe86e4c6aef9e1f42f 2023-04-08 02:54:51,986 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:51,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922551985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:51,992 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/bb522fe7018b4d419c11527865976620 2023-04-08 02:54:51,996 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/31774130daf2498b93908415de5fc15f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/31774130daf2498b93908415de5fc15f 2023-04-08 02:54:52,000 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/31774130daf2498b93908415de5fc15f, entries=150, sequenceid=203, filesize=11.8 K 2023-04-08 02:54:52,001 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/95cb51ce57f24bbe86e4c6aef9e1f42f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/95cb51ce57f24bbe86e4c6aef9e1f42f 2023-04-08 02:54:52,005 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/95cb51ce57f24bbe86e4c6aef9e1f42f, entries=150, sequenceid=203, filesize=11.8 K 2023-04-08 02:54:52,006 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/bb522fe7018b4d419c11527865976620 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/bb522fe7018b4d419c11527865976620 2023-04-08 02:54:52,009 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/bb522fe7018b4d419c11527865976620, entries=150, sequenceid=203, filesize=11.8 K 2023-04-08 02:54:52,009 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for d3e62a2534068dd21a6b490550cac850 in 451ms, sequenceid=203, compaction requested=true 2023-04-08 02:54:52,009 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:52,010 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:52,010 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:52,010 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:52,010 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,010 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:52,010 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:52,011 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,011 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:52,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,012 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:52,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:52,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:52,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:52,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:52,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:52,013 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@753073ab[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:52,013 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:52,013 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,014 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,014 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,014 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:52,014 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:52,014 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:52,014 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,014 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:52,016 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,016 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:52,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:52,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:52,016 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:52,017 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:52,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:52,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:52,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:52,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:52,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:52,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:52,019 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:52,019 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:52,019 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:52,019 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:52,019 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:52,021 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,021 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:52,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,021 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:52,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:52,022 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:52,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:52,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:52,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:52,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:52,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:52,023 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:52,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,025 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,025 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:52,025 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:52,025 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:52,025 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:52,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:52,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:52,025 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,026 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,026 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,026 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,026 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:52,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:52,025 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:52,026 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:52,026 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:52,026 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:52,026 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:52,126 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:52,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:52,128 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:52,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:52,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:52,129 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:52,129 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:52,129 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:52,129 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:52,130 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,130 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:52,131 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,131 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:52,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:52,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:52,131 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,131 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:52,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:52,132 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:52,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:52,132 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:52,132 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:52,132 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:52,132 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:52,132 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:52,132 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:52,132 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:52,133 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:52,133 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:52,133 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:52,133 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-08 02:54:52,133 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:54:52,133 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:52,133 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:54:52,133 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:52,133 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:54:52,133 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:52,143 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/1162f92f7db44779bacdd68e5831c608 2023-04-08 02:54:52,283 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:52,283 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:52,302 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:52,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922552299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,303 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:52,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922552299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,303 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:52,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922552299, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,303 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:52,303 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922552300, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,405 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:52,405 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922552404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:52,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922552404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:52,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922552404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,406 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:52,406 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922552404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,439 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:52,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922552438, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,564 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/2e4118c9fcb1494d8b29d59040b6255a 2023-04-08 02:54:52,583 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=219 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/b2bfc5602d0a4a508049209cfc218bb0 2023-04-08 02:54:52,589 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/1162f92f7db44779bacdd68e5831c608 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/1162f92f7db44779bacdd68e5831c608 2023-04-08 02:54:52,598 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/1162f92f7db44779bacdd68e5831c608, entries=150, sequenceid=219, filesize=11.8 K 2023-04-08 02:54:52,599 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/2e4118c9fcb1494d8b29d59040b6255a as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/2e4118c9fcb1494d8b29d59040b6255a 2023-04-08 02:54:52,603 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/2e4118c9fcb1494d8b29d59040b6255a, entries=150, sequenceid=219, filesize=11.8 K 2023-04-08 02:54:52,604 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/b2bfc5602d0a4a508049209cfc218bb0 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/b2bfc5602d0a4a508049209cfc218bb0 2023-04-08 02:54:52,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:52,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922552607, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,609 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/b2bfc5602d0a4a508049209cfc218bb0, entries=150, sequenceid=219, filesize=11.8 K 2023-04-08 02:54:52,610 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for d3e62a2534068dd21a6b490550cac850 in 477ms, sequenceid=219, compaction requested=true 2023-04-08 02:54:52,610 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:52,611 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:52,611 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:52,611 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:52,611 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,611 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:52,611 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:52,611 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-08 02:54:52,611 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:52,612 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:54:52,612 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:52,612 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:54:52,612 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:52,612 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:54:52,612 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:52,613 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:52,613 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:52,613 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,613 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,613 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:52,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:52,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:52,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:52,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:52,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:52,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@329b8c37[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:52,624 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:52,624 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,625 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,625 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:52,625 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:52,625 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:52,626 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,626 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:52,628 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/550ce511fbc747d59c977724737a9d3a 2023-04-08 02:54:52,628 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:52,629 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:52,628 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,629 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:52,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:52,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:52,629 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:52,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922552626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:52,629 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:52,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922552626, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:52,630 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:52,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922552628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:52,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:52,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:52,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:52,634 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:52,634 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:52,634 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:52,634 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:52,634 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:52,636 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,636 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:52,636 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:52,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,637 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:52,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:52,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:52,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:52,639 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:52,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:52,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,644 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,644 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:52,645 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:52,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:52,645 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:52,645 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:52,645 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:52,645 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:52,645 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:52,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:52,645 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:52,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:52,645 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,645 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,645 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,646 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:52,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:52,731 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:52,731 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922552730, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:52,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922552731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,733 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:52,733 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922552732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,745 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:52,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:52,747 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:52,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:52,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:52,748 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:52,748 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:52,748 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:52,748 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:52,749 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,749 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:52,750 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,750 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:52,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:52,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:52,750 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,750 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:52,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:52,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:52,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:52,751 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:52,751 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:52,751 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:52,751 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:52,751 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:52,751 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:52,752 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:52,752 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:52,752 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:52,752 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:52,752 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:52,752 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:52,752 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:52,752 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:52,752 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,752 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:52,752 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:52,754 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,754 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:52,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:52,754 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,755 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:52,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:52,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:52,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:52,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@311d46a0[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:52,756 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:52,756 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,757 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,757 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:52,757 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:52,757 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:52,757 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,758 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:52,758 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,758 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:52,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:52,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:52,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:52,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:52,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:52,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:52,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:52,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:52,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:52,761 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:52,761 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:52,761 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:52,761 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:52,761 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:52,762 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,762 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:52,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,763 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:52,763 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:52,763 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:52,764 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:52,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:52,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,770 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:52,770 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:52,770 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:52,770 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:52,770 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,770 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:52,771 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:52,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:52,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:52,770 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:52,771 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:52,771 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:52,771 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,771 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:52,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:52,771 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:52,771 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,771 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,871 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:52,871 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:52,873 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:52,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:52,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:52,874 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:52,874 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:52,874 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:52,874 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:52,875 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,875 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:52,876 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:52,876 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:52,876 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,876 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:52,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:52,877 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:52,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:52,877 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:52,877 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:52,877 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:52,877 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:52,877 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:52,878 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:52,878 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:52,878 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:52,878 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:52,878 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:52,878 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:52,878 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:52,878 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:52,878 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:52,878 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,878 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:52,878 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:52,880 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,880 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:52,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:52,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:52,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:52,881 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,881 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:52,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:52,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:52,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:52,881 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4aa4794a[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:52,881 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:52,882 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,883 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,883 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:52,883 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:52,883 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:52,883 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,883 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:52,888 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,888 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:52,888 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:52,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:52,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:52,888 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:52,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:52,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:52,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:52,890 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:52,890 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:52,890 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:52,890 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:52,890 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:52,890 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:52,892 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,892 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:52,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:52,892 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:52,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:52,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:52,892 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:52,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:52,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:52,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:52,893 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:52,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:52,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,896 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,896 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:52,896 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:52,896 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,896 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:52,896 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:52,896 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:52,896 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:52,896 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:52,896 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:52,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:52,896 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:52,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:52,896 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:52,896 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:52,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:52,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:52,909 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:52,909 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922552908, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,935 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:52,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922552933, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,936 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:52,936 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922552934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,937 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:52,937 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922552935, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:52,996 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:52,997 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:52,999 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:53,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:53,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:53,000 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:53,000 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:53,000 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:53,000 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:53,000 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,000 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:53,002 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:53,002 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:53,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:53,002 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,002 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:53,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:53,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:53,003 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:53,003 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:53,003 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:53,003 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:53,003 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:53,003 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:53,003 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:53,003 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:53,003 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:53,003 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:53,003 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:53,004 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:53,004 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:53,004 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:53,004 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:53,004 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,004 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:53,004 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:53,005 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,005 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:53,005 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,005 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:53,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:53,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:53,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:53,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:53,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:53,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:53,007 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cfff69e[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:53,007 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:53,007 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,008 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,008 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,008 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:53,008 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,009 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:53,008 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:53,009 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:53,011 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,011 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:53,011 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:53,011 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:53,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:53,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:53,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:53,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:53,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:53,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:53,015 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:53,015 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:53,015 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:53,015 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:53,015 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:53,016 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,017 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:53,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,017 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:53,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:53,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:53,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:53,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:53,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:53,019 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,020 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:53,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:53,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,024 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:53,024 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:53,024 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:53,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:53,024 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:53,024 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:53,024 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,024 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,024 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,024 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,025 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:53,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:53,025 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:53,025 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:53,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:53,025 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:53,025 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:53,045 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/9645eea91b3f4654a456645adca380be 2023-04-08 02:54:53,125 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:53,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:53,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:53,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:53,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:53,131 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:53,131 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:53,131 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:53,131 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:53,131 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,131 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:53,132 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,132 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:53,132 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:53,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:53,132 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,133 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:53,133 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:53,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:53,133 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:53,133 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:53,134 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:53,134 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:53,134 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:53,134 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:53,134 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:53,134 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:53,134 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:53,134 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:53,134 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:53,134 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:53,134 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:53,134 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:53,134 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,134 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:53,134 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:53,137 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,137 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:53,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:53,137 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,137 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:53,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:53,137 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:53,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:53,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:53,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:53,138 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@526b2f02[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:53,138 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:53,138 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,139 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,140 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,140 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:53,140 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:53,140 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:53,140 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,140 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:53,141 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:53,141 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:53,141 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,141 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:53,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:53,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:53,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:53,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:53,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:53,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:53,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:53,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:53,143 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:53,143 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:53,143 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:53,143 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:53,144 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:53,144 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:53,145 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,145 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,145 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:53,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:53,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:53,145 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:53,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:53,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:53,147 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:53,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:53,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,150 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,150 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:53,150 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:53,150 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:53,150 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:53,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:53,150 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:53,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:53,150 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:53,150 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:53,151 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:53,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:53,151 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,151 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:53,151 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,151 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,151 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:53,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:53,239 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:53,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922553238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:53,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922553238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,240 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:53,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922553238, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,251 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:53,251 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:53,253 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:53,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:53,254 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:53,254 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:53,254 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:53,254 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:53,254 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:53,255 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,255 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:53,256 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:53,256 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:53,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:53,256 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:53,256 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:53,256 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:53,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:53,257 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:53,257 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:53,257 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:53,257 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:53,257 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:53,257 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:53,264 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:53,264 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:53,265 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:53,265 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:53,265 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:53,265 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:53,265 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:53,265 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:53,265 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,265 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:53,265 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:53,267 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,267 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:53,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:53,267 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,267 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:53,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:53,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:53,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:53,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:53,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:53,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9985ba9[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:53,269 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:53,269 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,270 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,270 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:53,270 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:53,270 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:53,270 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,270 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:53,274 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,274 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:53,274 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:53,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:53,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:53,274 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:53,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:53,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:53,277 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:53,277 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:53,277 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:53,277 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:53,277 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:53,277 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:53,278 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,278 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:53,278 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,278 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:53,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:53,279 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:53,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:53,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,279 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:53,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:53,280 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:53,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,284 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:53,284 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:53,284 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,284 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:53,284 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:53,284 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:53,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:53,285 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:53,285 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:53,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:53,284 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:53,285 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,285 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,285 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,285 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,285 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:53,286 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:53,286 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:53,386 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:53,386 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:53,388 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:53,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:53,389 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:53,389 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:53,389 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:53,390 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:53,390 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:53,390 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,390 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:53,391 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,391 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:53,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:53,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:53,391 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,391 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:53,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:53,392 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:53,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:53,393 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:53,393 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:53,393 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:53,393 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:53,393 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:53,393 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:53,393 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:53,393 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:53,393 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:53,393 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:53,393 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:53,394 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:53,394 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:53,394 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:53,394 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,394 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:53,394 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:53,395 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,395 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:53,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:53,395 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,395 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:53,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:53,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:53,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@609f1df9[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:53,397 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:53,397 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,399 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,399 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:53,400 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:53,400 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:53,400 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,400 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:53,401 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,401 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:53,401 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:53,401 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:53,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:53,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:53,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:53,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:53,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:53,403 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:53,403 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:53,403 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:53,403 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:53,403 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:53,403 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:53,405 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,405 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,405 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:53,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:53,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:53,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:53,406 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:53,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:53,407 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:53,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:53,409 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:53,409 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:53,409 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:53,409 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,409 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:53,409 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:53,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:53,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:53,410 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:53,410 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:53,409 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:53,409 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:53,410 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,410 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,410 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:53,410 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,410 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:53,412 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:53,412 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:53,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:53,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922553410, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,457 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=238 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/a2c687c7a1f8419380ab5c91b7b5f6b8 2023-04-08 02:54:53,461 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/550ce511fbc747d59c977724737a9d3a as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/550ce511fbc747d59c977724737a9d3a 2023-04-08 02:54:53,467 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/550ce511fbc747d59c977724737a9d3a, entries=250, sequenceid=238, filesize=16.4 K 2023-04-08 02:54:53,468 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/9645eea91b3f4654a456645adca380be as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/9645eea91b3f4654a456645adca380be 2023-04-08 02:54:53,472 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/9645eea91b3f4654a456645adca380be, entries=150, sequenceid=238, filesize=11.8 K 2023-04-08 02:54:53,473 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/a2c687c7a1f8419380ab5c91b7b5f6b8 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/a2c687c7a1f8419380ab5c91b7b5f6b8 2023-04-08 02:54:53,477 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/a2c687c7a1f8419380ab5c91b7b5f6b8, entries=150, sequenceid=238, filesize=11.8 K 2023-04-08 02:54:53,478 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for d3e62a2534068dd21a6b490550cac850 in 867ms, sequenceid=238, compaction requested=true 2023-04-08 02:54:53,478 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:53,478 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:53,478 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 02:54:53,478 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 02:54:53,478 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:53,479 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:53,480 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 65302 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 02:54:53,480 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60522 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 02:54:53,480 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/A is initiating minor compaction (all files) 2023-04-08 02:54:53,480 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/B is initiating minor compaction (all files) 2023-04-08 02:54:53,480 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/A in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:53,480 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/B in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:53,481 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/e2418d2df4d44f5f86aa0805c657e8b6, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/ca1cc1c2640545e29c36a791ab2f91c8, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/31774130daf2498b93908415de5fc15f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/1162f92f7db44779bacdd68e5831c608, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/550ce511fbc747d59c977724737a9d3a] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=63.8 K 2023-04-08 02:54:53,481 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/51ba9f71b0eb4b71b7fad023bdc60caa, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/f54b65712bda43bdb983609884a3a536, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/95cb51ce57f24bbe86e4c6aef9e1f42f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/2e4118c9fcb1494d8b29d59040b6255a, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/9645eea91b3f4654a456645adca380be] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=59.1 K 2023-04-08 02:54:53,481 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting e2418d2df4d44f5f86aa0805c657e8b6, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680922490296 2023-04-08 02:54:53,481 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 51ba9f71b0eb4b71b7fad023bdc60caa, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680922490296 2023-04-08 02:54:53,481 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting ca1cc1c2640545e29c36a791ab2f91c8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1680922490917 2023-04-08 02:54:53,481 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting f54b65712bda43bdb983609884a3a536, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1680922490917 2023-04-08 02:54:53,482 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 31774130daf2498b93908415de5fc15f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680922491039 2023-04-08 02:54:53,482 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 95cb51ce57f24bbe86e4c6aef9e1f42f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680922491039 2023-04-08 02:54:53,482 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 1162f92f7db44779bacdd68e5831c608, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1680922491666 2023-04-08 02:54:53,482 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 2e4118c9fcb1494d8b29d59040b6255a, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1680922491666 2023-04-08 02:54:53,482 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 550ce511fbc747d59c977724737a9d3a, keycount=250, bloomtype=ROW, size=16.4 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680922492297 2023-04-08 02:54:53,483 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 9645eea91b3f4654a456645adca380be, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680922492297 2023-04-08 02:54:53,493 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#A#compaction#254 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:53,502 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#B#compaction#255 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:53,510 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:53,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:53,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:53,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:53,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:53,516 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:53,516 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:53,516 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:53,516 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:53,517 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,517 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:53,518 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:53,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:53,518 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:53,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:53,519 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,519 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:53,519 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:53,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:53,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:53,520 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:53,520 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:53,520 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:53,520 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:53,520 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:53,520 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:53,520 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:53,520 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:53,520 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:53,520 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:53,520 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-08 02:54:53,521 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:54:53,521 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:53,521 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:54:53,521 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:53,521 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:54:53,521 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:53,533 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/ffb80df32ef2492eb73d3ed962897d41 2023-04-08 02:54:53,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:53,744 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:53,762 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:53,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922553760, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,763 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:53,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922553761, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,764 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:53,764 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922553762, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,865 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:53,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922553863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,866 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:53,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 122 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922553864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:53,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922553865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:53,917 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/532fb819294f4cd1b6b75bd973ee406f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/532fb819294f4cd1b6b75bd973ee406f 2023-04-08 02:54:53,923 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/c2fc0a4d1c4c442984d96d85703a9119 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/c2fc0a4d1c4c442984d96d85703a9119 2023-04-08 02:54:53,924 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d3e62a2534068dd21a6b490550cac850/A of d3e62a2534068dd21a6b490550cac850 into 532fb819294f4cd1b6b75bd973ee406f(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:53,924 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:53,924 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/A, priority=11, startTime=1680922493478; duration=0sec 2023-04-08 02:54:53,924 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:53,924 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 02:54:53,926 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60522 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 02:54:53,926 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/C is initiating minor compaction (all files) 2023-04-08 02:54:53,926 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/C in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:53,926 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/48379221b677477bb49c122e4a792eb2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/b9d6edbc704c4e66b7abc835d4a06502, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/bb522fe7018b4d419c11527865976620, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/b2bfc5602d0a4a508049209cfc218bb0, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/a2c687c7a1f8419380ab5c91b7b5f6b8] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=59.1 K 2023-04-08 02:54:53,927 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 48379221b677477bb49c122e4a792eb2, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=168, earliestPutTs=1680922490296 2023-04-08 02:54:53,927 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting b9d6edbc704c4e66b7abc835d4a06502, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=184, earliestPutTs=1680922490917 2023-04-08 02:54:53,927 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting bb522fe7018b4d419c11527865976620, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680922491039 2023-04-08 02:54:53,927 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d3e62a2534068dd21a6b490550cac850/B of d3e62a2534068dd21a6b490550cac850 into c2fc0a4d1c4c442984d96d85703a9119(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:53,928 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:53,928 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/B, priority=11, startTime=1680922493478; duration=0sec 2023-04-08 02:54:53,928 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:53,928 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting b2bfc5602d0a4a508049209cfc218bb0, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=219, earliestPutTs=1680922491666 2023-04-08 02:54:53,928 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting a2c687c7a1f8419380ab5c91b7b5f6b8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680922492297 2023-04-08 02:54:53,936 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#C#compaction#258 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:53,945 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/d848cf5774054fbc9593b3124c7f705b 2023-04-08 02:54:54,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:54,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922554067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,069 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:54,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922554068, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,071 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:54,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922554070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,345 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/65f8e93f1003412da014622ab5513830 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/65f8e93f1003412da014622ab5513830 2023-04-08 02:54:54,349 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in d3e62a2534068dd21a6b490550cac850/C of d3e62a2534068dd21a6b490550cac850 into 65f8e93f1003412da014622ab5513830(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:54,349 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:54,349 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/C, priority=11, startTime=1680922493479; duration=0sec 2023-04-08 02:54:54,349 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:54,359 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=251 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/00e350c3b5614847bbcbf120a93bb663 2023-04-08 02:54:54,363 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/ffb80df32ef2492eb73d3ed962897d41 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/ffb80df32ef2492eb73d3ed962897d41 2023-04-08 02:54:54,366 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/ffb80df32ef2492eb73d3ed962897d41, entries=150, sequenceid=251, filesize=11.8 K 2023-04-08 02:54:54,367 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/d848cf5774054fbc9593b3124c7f705b as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d848cf5774054fbc9593b3124c7f705b 2023-04-08 02:54:54,370 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:54,370 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:54,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922554370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53102 deadline: 1680922554370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,372 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d848cf5774054fbc9593b3124c7f705b, entries=150, sequenceid=251, filesize=11.8 K 2023-04-08 02:54:54,372 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/00e350c3b5614847bbcbf120a93bb663 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/00e350c3b5614847bbcbf120a93bb663 2023-04-08 02:54:54,375 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:54,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922554374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,376 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/00e350c3b5614847bbcbf120a93bb663, entries=150, sequenceid=251, filesize=11.8 K 2023-04-08 02:54:54,377 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for d3e62a2534068dd21a6b490550cac850 in 857ms, sequenceid=251, compaction requested=false 2023-04-08 02:54:54,377 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:54,377 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:54,377 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:54,377 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:54,377 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,377 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:54,377 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:54,379 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,379 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:54,379 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,379 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:54,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:54,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:54,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:54,381 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@23975c72[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:54,381 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:54,381 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,388 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,388 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:54,388 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:54,388 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:54,389 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,389 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:54,390 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:54,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:54,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:54,390 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:54,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:54,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:54,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:54,394 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:54,394 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:54,394 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:54,394 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:54,394 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:54,394 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:54,396 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,396 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:54,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,396 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:54,397 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:54,397 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:54,399 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:54,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,401 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,401 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:54,401 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:54,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:54,401 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:54,401 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:54,402 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,402 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,402 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,402 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:54,402 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:54,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:54,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:54,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:54,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:54,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:54,402 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:54,402 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:54,415 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:54,415 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-08 02:54:54,415 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:54:54,415 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:54,415 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:54:54,415 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:54,415 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:54:54,415 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:54,425 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=274 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/2fbae948cce0453a995707498943d20c 2023-04-08 02:54:54,441 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:54,442 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922554440, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,455 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:54,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922554454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,457 DEBUG [Thread-1618] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4147 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:54:54,502 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:54,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:54,504 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:54,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:54,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:54,506 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:54,506 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:54,506 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:54,506 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:54,506 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,506 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:54,508 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,508 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:54,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:54,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:54,509 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,509 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:54,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:54,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:54,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:54,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:54,510 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:54,510 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:54,510 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:54,510 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:54,510 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:54,510 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:54,510 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:54,510 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:54,510 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:54,510 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:54,510 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:54,510 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:54,510 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:54,510 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,510 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:54,510 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:54,512 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,512 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:54,512 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,512 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:54,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:54,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:54,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@280f743d[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:54,514 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:54,514 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,515 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,515 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:54,515 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:54,515 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:54,515 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,515 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:54,516 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,516 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:54,516 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:54,517 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:54,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:54,519 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:54,519 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:54,519 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:54,519 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:54,519 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:54,519 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:54,521 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,521 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:54,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,521 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:54,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:54,522 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,523 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:54,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,528 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,528 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:54,528 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:54,528 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:54,528 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:54,528 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:54,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:54,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:54,528 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:54,528 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:54,528 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,528 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:54,528 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:54,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:54,528 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:54,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:54,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922554542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,628 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:54,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:54,630 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:54,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:54,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:54,632 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:54,632 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:54,632 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:54,632 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:54,632 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,632 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:54,633 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:54,633 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:54,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:54,634 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,634 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:54,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:54,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:54,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:54,634 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:54,635 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:54,635 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:54,635 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:54,635 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:54,635 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:54,635 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:54,635 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:54,635 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:54,635 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:54,635 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:54,635 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:54,635 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:54,635 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:54,635 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,635 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:54,635 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:54,637 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,637 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:54,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:54,637 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,637 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:54,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:54,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:54,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:54,638 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:54,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@63813786[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:54,639 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:54,639 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,641 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,641 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,641 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:54,641 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:54,641 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:54,641 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,641 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:54,643 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,643 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:54,643 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:54,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:54,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:54,643 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:54,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:54,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:54,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:54,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:54,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:54,645 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:54,645 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:54,645 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:54,645 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:54,645 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:54,647 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,647 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:54,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,647 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:54,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:54,647 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,649 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:54,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,650 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,650 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:54,650 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:54,651 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:54,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:54,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:54,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:54,651 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:54,651 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:54,651 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,651 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,651 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,651 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:54,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:54,651 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:54,651 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:54,651 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:54,746 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:54,746 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53172 deadline: 1680922554744, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,751 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:54,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:54,753 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:54,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:54,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:54,755 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:54,755 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:54,755 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:54,755 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:54,755 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,755 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:54,757 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:54,757 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:54,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:54,757 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,758 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:54,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:54,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:54,758 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:54,759 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:54,759 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:54,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:54,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:54,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:54,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:54,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:54,759 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:54,759 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:54,759 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:54,759 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:54,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:54,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:54,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:54,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:54,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,761 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:54,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:54,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:54,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:54,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7b699eda[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:54,763 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:54,763 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,764 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,764 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,764 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:54,764 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:54,764 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,764 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:54,764 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:54,766 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,766 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:54,766 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:54,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:54,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:54,766 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:54,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:54,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:54,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:54,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:54,769 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:54,769 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:54,769 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:54,769 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:54,769 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:54,769 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:54,776 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,776 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:54,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:54,776 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:54,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,777 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:54,777 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:54,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,778 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,780 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,780 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:54,780 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:54,780 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,780 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:54,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:54,780 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:54,781 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:54,781 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,781 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:54,781 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:54,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:54,781 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:54,781 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:54,781 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:54,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:54,840 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=274 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/8b8054e87cf547d597aba62d21354a6d 2023-04-08 02:54:54,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:54,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922554872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,872 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:54,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53102 deadline: 1680922554872, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,878 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:54,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922554878, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,880 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=274 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/c289668a5f0841bdade9fb51511e19c4 2023-04-08 02:54:54,881 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:54,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:54,885 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:54,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:54,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:54,886 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/2fbae948cce0453a995707498943d20c as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/2fbae948cce0453a995707498943d20c 2023-04-08 02:54:54,886 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:54,886 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:54,886 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:54,886 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:54,887 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,887 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:54,889 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:54,889 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:54,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:54,889 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:54,889 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,889 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:54,890 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:54,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:54,890 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:54,891 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:54,891 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:54,891 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:54,891 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:54,891 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:54,892 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:54,892 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:54,892 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:54,892 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:54,892 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:54,892 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:54,892 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:54,892 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:54,892 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,892 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:54,892 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:54,893 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/2fbae948cce0453a995707498943d20c, entries=200, sequenceid=274, filesize=14.3 K 2023-04-08 02:54:54,894 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,894 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:54,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:54,895 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/8b8054e87cf547d597aba62d21354a6d as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/8b8054e87cf547d597aba62d21354a6d 2023-04-08 02:54:54,895 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,895 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:54,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:54,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:54,897 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:54,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62843909[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:54,898 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:54,898 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,900 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,900 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:54,900 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:54,900 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:54,900 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,900 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:54,901 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:54,901 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:54,901 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:54,902 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:54,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:54,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:54,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:54,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:54,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:54,906 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:54,906 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:54,906 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:54,906 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:54,906 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:54,907 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,907 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,907 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:54,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:54,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:54,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,908 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/8b8054e87cf547d597aba62d21354a6d, entries=150, sequenceid=274, filesize=11.9 K 2023-04-08 02:54:54,909 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:54,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:54,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:54,909 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/c289668a5f0841bdade9fb51511e19c4 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/c289668a5f0841bdade9fb51511e19c4 2023-04-08 02:54:54,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:54,910 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:54,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:54,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,913 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/c289668a5f0841bdade9fb51511e19c4, entries=150, sequenceid=274, filesize=11.9 K 2023-04-08 02:54:54,913 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:54,913 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:54,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:54,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:54,914 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for d3e62a2534068dd21a6b490550cac850 in 499ms, sequenceid=274, compaction requested=true 2023-04-08 02:54:54,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:54,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:54,914 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:54,914 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:54,914 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:54,914 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:54:54,914 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:54:54,914 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:54,915 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36770 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:54:54,915 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/B is initiating minor compaction (all files) 2023-04-08 02:54:54,915 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/B in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:54,915 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/c2fc0a4d1c4c442984d96d85703a9119, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d848cf5774054fbc9593b3124c7f705b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/8b8054e87cf547d597aba62d21354a6d] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=35.9 K 2023-04-08 02:54:54,915 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39210 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:54:54,916 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/A is initiating minor compaction (all files) 2023-04-08 02:54:54,916 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/A in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:54,916 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/532fb819294f4cd1b6b75bd973ee406f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/ffb80df32ef2492eb73d3ed962897d41, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/2fbae948cce0453a995707498943d20c] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=38.3 K 2023-04-08 02:54:54,916 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting c2fc0a4d1c4c442984d96d85703a9119, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680922492297 2023-04-08 02:54:54,916 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,916 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:54,916 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:54,916 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:54,916 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:54,916 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,916 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:54,916 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:54,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:54,916 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:54,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:54,916 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 532fb819294f4cd1b6b75bd973ee406f, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680922492297 2023-04-08 02:54:54,917 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:54,917 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:54,917 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting d848cf5774054fbc9593b3124c7f705b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680922492621 2023-04-08 02:54:54,917 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting ffb80df32ef2492eb73d3ed962897d41, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680922492621 2023-04-08 02:54:54,917 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 8b8054e87cf547d597aba62d21354a6d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1680922493754 2023-04-08 02:54:54,917 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 2fbae948cce0453a995707498943d20c, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1680922493754 2023-04-08 02:54:54,925 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#A#compaction#263 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:54,925 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#B#compaction#264 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:54,934 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/8ae1e7a646614d41b7a478efa1e028b9 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/8ae1e7a646614d41b7a478efa1e028b9 2023-04-08 02:54:54,939 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/A of d3e62a2534068dd21a6b490550cac850 into 8ae1e7a646614d41b7a478efa1e028b9(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:54,939 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:54,939 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/A, priority=13, startTime=1680922494914; duration=0sec 2023-04-08 02:54:54,939 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:54,939 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:54:54,940 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36770 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:54:54,940 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/C is initiating minor compaction (all files) 2023-04-08 02:54:54,940 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/C in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:54,940 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/65f8e93f1003412da014622ab5513830, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/00e350c3b5614847bbcbf120a93bb663, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/c289668a5f0841bdade9fb51511e19c4] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=35.9 K 2023-04-08 02:54:54,940 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 65f8e93f1003412da014622ab5513830, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=238, earliestPutTs=1680922492297 2023-04-08 02:54:54,941 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 00e350c3b5614847bbcbf120a93bb663, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=251, earliestPutTs=1680922492621 2023-04-08 02:54:54,941 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting c289668a5f0841bdade9fb51511e19c4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1680922493754 2023-04-08 02:54:54,946 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#C#compaction#265 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:55,017 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:55,017 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:55,019 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:55,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:55,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:55,024 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:55,024 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:55,024 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:55,024 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:55,024 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,025 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,029 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:55,029 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,029 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:55,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:55,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,030 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,030 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:55,030 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:55,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,030 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:55,031 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:55,031 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:55,031 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:55,031 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:55,031 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:55,032 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:55,032 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:55,032 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:55,032 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:55,032 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-08 02:54:55,032 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:54:55,032 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:55,033 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:54:55,033 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:55,033 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:54:55,033 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:55,043 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/a4235e0431d84c66923c515fde6ed748 2023-04-08 02:54:55,050 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:55,050 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:55,055 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/e4eb91ade5ee4e52bbf1a699c7162103 2023-04-08 02:54:55,072 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/aa98a0988bab4340a44f52761f2d2b62 2023-04-08 02:54:55,077 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/a4235e0431d84c66923c515fde6ed748 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/a4235e0431d84c66923c515fde6ed748 2023-04-08 02:54:55,081 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/a4235e0431d84c66923c515fde6ed748, entries=150, sequenceid=287, filesize=11.9 K 2023-04-08 02:54:55,081 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/e4eb91ade5ee4e52bbf1a699c7162103 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e4eb91ade5ee4e52bbf1a699c7162103 2023-04-08 02:54:55,085 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e4eb91ade5ee4e52bbf1a699c7162103, entries=150, sequenceid=287, filesize=11.9 K 2023-04-08 02:54:55,086 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/aa98a0988bab4340a44f52761f2d2b62 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/aa98a0988bab4340a44f52761f2d2b62 2023-04-08 02:54:55,089 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/aa98a0988bab4340a44f52761f2d2b62, entries=150, sequenceid=287, filesize=11.9 K 2023-04-08 02:54:55,089 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=80.51 KB/82440 for d3e62a2534068dd21a6b490550cac850 in 57ms, sequenceid=287, compaction requested=false 2023-04-08 02:54:55,089 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:55,089 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:55,089 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:55,089 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:55,089 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,090 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:55,090 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:55,091 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,091 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:55,091 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,091 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:55,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:55,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:55,093 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-08 02:54:55,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:55,093 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31f907f8[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:55,093 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:54:55,093 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:55,093 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:55,093 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,093 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:54:55,093 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:55,093 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:54:55,093 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:55,096 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,096 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:55,096 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:55,096 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:55,096 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,096 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:55,098 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,098 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:55,098 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:55,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:55,099 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:55,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:55,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:55,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:55,101 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:55,101 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:55,101 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:55,101 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:55,101 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:55,103 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,103 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,103 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:55,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:55,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:55,103 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:55,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,105 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:55,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,107 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:55,107 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,107 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,107 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,107 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,107 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,107 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,107 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:55,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:55,107 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:55,107 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:55,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:55,107 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:55,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:55,108 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:55,108 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:55,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:55,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:55,129 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:55,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53172 deadline: 1680922555127, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,208 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:55,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:55,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:55,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:55,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:55,211 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:55,211 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:55,211 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:55,212 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:55,212 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,212 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,213 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:55,213 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:55,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:55,213 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,213 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,214 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:55,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,214 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:55,215 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:55,215 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:55,215 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:55,215 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:55,215 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:55,215 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:55,215 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:55,215 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:55,215 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:55,215 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:55,215 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:55,215 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:55,215 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,216 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:55,216 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:55,220 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:55,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:55,220 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:55,220 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,221 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:55,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:55,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:55,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:55,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bae2b46[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:55,222 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:55,222 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,223 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,223 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:55,223 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,223 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:55,223 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:55,223 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:55,224 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:55,224 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:55,225 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,225 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:55,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:55,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:55,227 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:55,227 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:55,227 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:55,227 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:55,227 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:55,227 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:55,229 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,229 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:55,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,229 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:55,229 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:55,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:55,230 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:55,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,231 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:55,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,232 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:55,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53172 deadline: 1680922555230, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,234 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,234 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,234 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,234 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:55,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:55,234 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:55,234 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:55,234 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:55,234 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:55,234 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,234 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:55,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:55,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:55,234 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:55,234 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,235 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,235 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:55,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:55,335 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:55,335 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/aa9e7cedcdf843288ed7827bc12bdc56 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/aa9e7cedcdf843288ed7827bc12bdc56 2023-04-08 02:54:55,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:55,337 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:55,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:55,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:55,338 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:55,338 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:55,338 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:55,338 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:55,339 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,339 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,340 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/B of d3e62a2534068dd21a6b490550cac850 into aa9e7cedcdf843288ed7827bc12bdc56(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:55,340 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:55,340 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/B, priority=13, startTime=1680922494914; duration=0sec 2023-04-08 02:54:55,340 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:55,340 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:55,340 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,340 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:55,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:55,341 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,341 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,341 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:55,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,341 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:55,342 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:55,342 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:55,342 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:55,342 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:55,342 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:55,342 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:55,342 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:55,342 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:55,342 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:55,342 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:55,342 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:55,342 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:55,342 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:55,342 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,342 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:55,342 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:55,344 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,344 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:55,344 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,344 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:55,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:55,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:55,346 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:55,346 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c86e1a7[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:55,346 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,347 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,347 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:55,347 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:55,347 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,347 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:55,347 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:55,349 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,349 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:55,349 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:55,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:55,349 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:55,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:55,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:55,352 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:55,353 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:55,353 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:55,353 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:55,353 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:55,354 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,354 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:55,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,355 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:55,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:55,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:55,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:55,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,355 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/98f2a05a2a78439ba93562cc4dc4b5b0 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/98f2a05a2a78439ba93562cc4dc4b5b0 2023-04-08 02:54:55,356 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:55,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:55,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,359 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:55,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,361 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/C of d3e62a2534068dd21a6b490550cac850 into 98f2a05a2a78439ba93562cc4dc4b5b0(size=12.5 K), total size for store is 24.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:55,361 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:55,361 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/C, priority=13, startTime=1680922494914; duration=0sec 2023-04-08 02:54:55,361 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:55,361 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:55,361 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:55,361 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:55,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:55,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:55,361 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,361 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:55,361 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,361 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:55,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:55,362 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,362 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,362 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,362 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,362 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:55,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:55,362 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:55,362 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:55,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:55,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53172 deadline: 1680922555433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,462 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:55,463 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:55,464 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:55,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:55,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:55,465 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:55,465 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:55,466 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:55,466 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:55,466 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,466 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,467 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:55,467 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:55,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:55,467 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,468 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,468 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:55,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,468 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:55,469 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:55,469 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:55,469 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:55,469 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:55,469 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:55,469 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:55,469 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:55,469 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:55,469 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:55,469 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:55,469 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:55,469 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:55,469 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:55,469 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,469 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:55,469 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:55,474 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,474 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:55,475 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,475 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:55,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:55,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:55,476 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2f89d14f[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:55,476 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:55,476 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,477 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,477 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:55,477 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:55,477 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:55,477 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,477 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:55,478 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:55,478 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,479 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:55,479 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:55,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:55,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:55,481 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:55,481 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:55,481 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:55,481 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:55,481 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:55,481 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:55,483 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,483 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,483 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:55,483 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:55,484 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:55,484 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:55,485 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:55,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,487 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:55,487 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:55,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:55,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:55,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:55,487 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:55,487 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:55,487 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,487 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:55,487 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:55,487 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,487 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:55,487 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:55,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:55,487 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,488 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,514 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/cf400ef63737442588973f2dc11d7be9 2023-04-08 02:54:55,587 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:55,588 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:55,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:55,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:55,590 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:55,590 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:55,590 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:55,590 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:55,590 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:55,591 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,591 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,592 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:55,592 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:55,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:55,592 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,593 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:55,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,593 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:55,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,594 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:55,594 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:55,594 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:55,594 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:55,594 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:55,594 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:55,594 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:55,594 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:55,595 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:55,595 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:55,595 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:55,595 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:55,595 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:55,595 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:55,595 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,595 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:55,595 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:55,596 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,596 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:55,596 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,596 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:55,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:55,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:55,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:55,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:55,598 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5636d581[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:55,598 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:55,598 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,600 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,600 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,600 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:55,600 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:55,600 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,600 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:55,600 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:55,601 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:55,601 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,601 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:55,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:55,601 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:55,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:55,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:55,604 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:55,604 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:55,604 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:55,604 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:55,604 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:55,604 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:55,605 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,605 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,605 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:55,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:55,606 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:55,606 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,607 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,609 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,609 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:55,609 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,609 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:55,609 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:55,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:55,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:55,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:55,609 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:55,609 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:55,609 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:55,610 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:55,610 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:55,610 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:55,610 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,610 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,610 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,710 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:55,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:55,712 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:55,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:55,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:55,714 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:55,714 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:55,714 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:55,714 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:55,715 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,715 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,720 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:55,720 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:55,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:55,720 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,720 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:55,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,721 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,721 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:55,736 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:55,736 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:55,736 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:55,736 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:55,736 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:55,736 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:55,736 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:55,736 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:55,736 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:55,737 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:55,737 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:55,737 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:55,737 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:55,737 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,737 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:55,737 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:55,738 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:55,738 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53172 deadline: 1680922555737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,739 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:55,739 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:55,739 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,739 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:55,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:55,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:55,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:55,740 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c68301a[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:55,740 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:55,740 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,741 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,742 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,742 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:55,742 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,742 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:55,742 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:55,742 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:55,744 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,744 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:55,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:55,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:55,744 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:55,744 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:55,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:55,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:55,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:55,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:55,746 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:55,746 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:55,746 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:55,746 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:55,747 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:55,747 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:55,748 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,748 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,748 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:55,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:55,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:55,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:55,749 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:55,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,750 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:55,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,752 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,752 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,752 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:55,752 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:55,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:55,752 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,752 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,752 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,752 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,752 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:55,752 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:55,752 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:55,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:55,752 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:55,753 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:55,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:55,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:55,853 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:55,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:55,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:55,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:55,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:55,857 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:55,857 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:55,857 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:55,857 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:55,857 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,857 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,860 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:55,860 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:55,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:55,860 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,860 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:55,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,861 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:55,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:55,862 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:55,862 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:55,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:55,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:55,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:55,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:55,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:55,862 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:55,862 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:55,862 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:55,862 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:55,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:55,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:55,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:55,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:55,864 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,864 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:55,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:55,864 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,864 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:55,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:55,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:55,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27b87de9[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:55,866 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:55,866 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,867 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,867 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,867 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,867 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:55,867 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:55,867 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:55,867 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,867 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:55,872 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:55,872 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,872 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:55,872 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:55,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:55,873 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:55,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:55,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:55,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:55,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:55,875 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:55,875 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:55,875 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:55,875 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:55,875 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:55,876 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,877 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:55,877 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,877 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:55,877 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:55,879 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:55,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,881 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:55,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922555879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,882 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:55,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53102 deadline: 1680922555881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,882 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,882 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,882 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,882 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:55,882 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,882 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:55,882 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,882 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:55,882 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:55,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:55,883 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:55,882 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:55,883 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:55,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:55,883 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:55,883 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:55,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:55,885 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53122 deadline: 1680922555883, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,924 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/06eaf40d3f234b85b12a8342cab4a925 2023-04-08 02:54:55,943 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=303 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/36c9f184e80944d9ac3f590dd4978e1f 2023-04-08 02:54:55,949 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/cf400ef63737442588973f2dc11d7be9 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/cf400ef63737442588973f2dc11d7be9 2023-04-08 02:54:55,954 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/cf400ef63737442588973f2dc11d7be9, entries=150, sequenceid=303, filesize=11.9 K 2023-04-08 02:54:55,956 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/06eaf40d3f234b85b12a8342cab4a925 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/06eaf40d3f234b85b12a8342cab4a925 2023-04-08 02:54:55,959 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/06eaf40d3f234b85b12a8342cab4a925, entries=150, sequenceid=303, filesize=11.9 K 2023-04-08 02:54:55,960 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/36c9f184e80944d9ac3f590dd4978e1f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/36c9f184e80944d9ac3f590dd4978e1f 2023-04-08 02:54:55,964 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/36c9f184e80944d9ac3f590dd4978e1f, entries=150, sequenceid=303, filesize=11.9 K 2023-04-08 02:54:55,965 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for d3e62a2534068dd21a6b490550cac850 in 872ms, sequenceid=303, compaction requested=true 2023-04-08 02:54:55,965 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:55,965 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:55,965 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:54:55,965 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:55,965 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:54:55,965 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:55,966 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:54:55,966 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/A is initiating minor compaction (all files) 2023-04-08 02:54:55,966 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:54:55,966 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/A in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:55,966 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/B is initiating minor compaction (all files) 2023-04-08 02:54:55,967 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/8ae1e7a646614d41b7a478efa1e028b9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/a4235e0431d84c66923c515fde6ed748, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/cf400ef63737442588973f2dc11d7be9] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=36.3 K 2023-04-08 02:54:55,967 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/B in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:55,967 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/aa9e7cedcdf843288ed7827bc12bdc56, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e4eb91ade5ee4e52bbf1a699c7162103, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/06eaf40d3f234b85b12a8342cab4a925] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=36.3 K 2023-04-08 02:54:55,967 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 8ae1e7a646614d41b7a478efa1e028b9, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1680922493754 2023-04-08 02:54:55,967 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting aa9e7cedcdf843288ed7827bc12bdc56, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1680922493754 2023-04-08 02:54:55,967 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting a4235e0431d84c66923c515fde6ed748, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1680922494417 2023-04-08 02:54:55,967 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting e4eb91ade5ee4e52bbf1a699c7162103, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1680922494417 2023-04-08 02:54:55,968 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting cf400ef63737442588973f2dc11d7be9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1680922495060 2023-04-08 02:54:55,968 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 06eaf40d3f234b85b12a8342cab4a925, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1680922495060 2023-04-08 02:54:55,974 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#A#compaction#272 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:55,975 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#B#compaction#273 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:55,983 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:55,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:55,986 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/ad50f33edc224fca94917315d4681bb1 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/ad50f33edc224fca94917315d4681bb1 2023-04-08 02:54:55,989 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/081f5868ffb9469b96ce4c11b3ad27a6 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/081f5868ffb9469b96ce4c11b3ad27a6 2023-04-08 02:54:55,992 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/B of d3e62a2534068dd21a6b490550cac850 into ad50f33edc224fca94917315d4681bb1(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:55,992 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:55,992 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/B, priority=13, startTime=1680922495965; duration=0sec 2023-04-08 02:54:55,992 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:55,992 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:54:55,992 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:55,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:55,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:55,994 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:55,994 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:55,994 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:55,994 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:55,994 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,994 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,995 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,995 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:55,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:55,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:55,995 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:55,996 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:55,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,996 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:55,996 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37172 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:54:55,996 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/C is initiating minor compaction (all files) 2023-04-08 02:54:55,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:55,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:55,996 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:55,996 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/C in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:55,997 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:55,997 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:55,997 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/98f2a05a2a78439ba93562cc4dc4b5b0, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/aa98a0988bab4340a44f52761f2d2b62, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/36c9f184e80944d9ac3f590dd4978e1f] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=36.3 K 2023-04-08 02:54:55,997 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:55,997 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:55,997 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:55,997 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:55,997 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/A of d3e62a2534068dd21a6b490550cac850 into 081f5868ffb9469b96ce4c11b3ad27a6(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:55,997 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 98f2a05a2a78439ba93562cc4dc4b5b0, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=274, earliestPutTs=1680922493754 2023-04-08 02:54:55,997 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:55,997 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:55,997 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:55,997 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:55,997 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/A, priority=13, startTime=1680922495965; duration=0sec 2023-04-08 02:54:55,998 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting aa98a0988bab4340a44f52761f2d2b62, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1680922494417 2023-04-08 02:54:55,998 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:55,998 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-08 02:54:55,998 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 36c9f184e80944d9ac3f590dd4978e1f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1680922495060 2023-04-08 02:54:55,998 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:54:55,998 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:55,998 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:54:55,999 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:55,999 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:54:55,999 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:56,004 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#C#compaction#275 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:56,016 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/4a8ca4d0a9c5476d920418f9418b4f8e 2023-04-08 02:54:56,026 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/2500fd73318b401dab88b03f7ef73e1d 2023-04-08 02:54:56,036 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=323 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/f635ca9c9f7e4f488eda116355d196ad 2023-04-08 02:54:56,040 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/4a8ca4d0a9c5476d920418f9418b4f8e as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/4a8ca4d0a9c5476d920418f9418b4f8e 2023-04-08 02:54:56,044 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/4a8ca4d0a9c5476d920418f9418b4f8e, entries=150, sequenceid=323, filesize=11.9 K 2023-04-08 02:54:56,045 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/2500fd73318b401dab88b03f7ef73e1d as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/2500fd73318b401dab88b03f7ef73e1d 2023-04-08 02:54:56,048 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/2500fd73318b401dab88b03f7ef73e1d, entries=150, sequenceid=323, filesize=11.9 K 2023-04-08 02:54:56,049 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/f635ca9c9f7e4f488eda116355d196ad as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/f635ca9c9f7e4f488eda116355d196ad 2023-04-08 02:54:56,052 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/f635ca9c9f7e4f488eda116355d196ad, entries=150, sequenceid=323, filesize=11.9 K 2023-04-08 02:54:56,053 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=0 B/0 for d3e62a2534068dd21a6b490550cac850 in 55ms, sequenceid=323, compaction requested=false 2023-04-08 02:54:56,053 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:56,053 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:56,053 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:56,053 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:56,053 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,053 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:56,053 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:56,055 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,056 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:56,056 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,056 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:56,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:56,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:56,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46dac3f8[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:56,057 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:56,058 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,059 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,059 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:56,059 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:56,059 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:56,059 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,059 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:56,060 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,060 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:56,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:56,060 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:56,060 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:56,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:56,063 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:56,063 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:56,063 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:56,063 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:56,063 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:56,063 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:56,065 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,065 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,065 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:56,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:56,065 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:56,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,067 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:56,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,069 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:56,070 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:56,070 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,070 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,070 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:56,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:56,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:56,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:56,070 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,070 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,070 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,070 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,070 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:56,070 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:56,070 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:56,070 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:56,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:56,170 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:56,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:56,172 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:56,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:56,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:56,174 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:56,174 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:56,174 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:56,174 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:56,174 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,174 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,175 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:56,175 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,175 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:56,176 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,176 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:56,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,176 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:56,177 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:56,177 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:56,177 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:56,177 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:56,177 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:56,177 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:56,177 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:56,178 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,178 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:56,178 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:56,179 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,179 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:56,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:56,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,180 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:56,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:56,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:56,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:56,181 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26a09c7f[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:56,181 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:56,181 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,182 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,182 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:56,182 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:56,183 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,183 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:56,184 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:56,184 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:56,185 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:56,185 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:56,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:56,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:56,187 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:56,187 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:56,187 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:56,187 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:56,187 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:56,189 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,189 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,189 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:56,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:56,189 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,190 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,193 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:56,193 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:56,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:56,193 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,193 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,193 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,193 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:56,193 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:56,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:56,193 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:56,193 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,193 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,193 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,193 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:56,193 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:56,193 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,193 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:56,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:56,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:56,255 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-08 02:54:56,255 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:54:56,255 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:56,255 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:54:56,255 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:56,255 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:54:56,255 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:56,263 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/10c38b75155a45dba721e52823c9e635 2023-04-08 02:54:56,273 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/98cd3f7a91d74f1a827f621c9712504e 2023-04-08 02:54:56,294 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:56,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:56,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:56,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922496273, max timestamp=1680922496298Num uniques 859; 2023-04-08 02:54:56,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922496273, max timestamp=1680922496298Num uniques 859; 2023-04-08 02:54:56,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:54:56,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:54:56,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922496273, max timestamp=1680922496298Num uniques 859; 2023-04-08 02:54:56,300 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:54:56,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:56,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:56,304 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:56,304 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:56,304 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:56,304 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:56,304 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,304 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,306 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,306 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:56,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:56,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:56,306 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,306 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:56,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,307 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:56,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,307 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:56,307 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:56,307 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:56,307 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:56,307 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:56,308 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:56,308 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:56,308 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:56,308 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:56,308 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:56,308 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:56,308 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:56,308 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:56,308 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:56,308 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,308 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:56,308 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:56,311 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:56,312 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:56,312 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,312 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:56,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:56,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:56,315 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1bf59051[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:56,315 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:56,315 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,316 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,316 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:56,316 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:56,316 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:56,316 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,316 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:56,317 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,317 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:56,317 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:56,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:56,317 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:56,320 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:56,320 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:56,320 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:56,320 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:56,320 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:56,320 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:56,326 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,326 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:56,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:56,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,326 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:56,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:56,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:56,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:56,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,327 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:56,329 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,333 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:56,333 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:56,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:56,333 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:56,333 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:56,333 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:56,333 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,333 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,333 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,333 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,333 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,333 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:56,334 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:56,333 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:56,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:56,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:56,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:56,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53172 deadline: 1680922556360, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,413 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/2852ab4ca852432ebb55d93e507b02cc as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/2852ab4ca852432ebb55d93e507b02cc 2023-04-08 02:54:56,419 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/C of d3e62a2534068dd21a6b490550cac850 into 2852ab4ca852432ebb55d93e507b02cc(size=12.6 K), total size for store is 24.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:56,419 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:56,419 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/C, priority=13, startTime=1680922495965; duration=0sec 2023-04-08 02:54:56,419 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:56,434 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:56,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:56,436 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:56,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:56,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:56,437 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:56,437 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:56,437 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:56,437 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:56,437 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,437 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,438 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,438 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:56,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:56,439 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,439 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,439 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:56,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,439 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:56,440 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:56,440 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:56,440 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:56,440 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:56,440 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:56,440 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:56,440 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:56,440 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:56,440 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:56,440 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:56,440 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:56,440 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:56,440 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:56,440 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,440 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:56,440 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:56,442 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,442 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:56,442 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,442 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:56,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:56,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:56,444 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7fc6e1f3[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:56,444 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:56,444 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,446 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,446 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,446 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,447 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:56,447 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:56,447 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:56,447 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:56,448 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:56,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:56,448 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:56,448 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:56,448 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:56,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:56,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:56,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:56,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:56,450 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:56,450 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:56,450 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:56,450 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:56,450 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:56,450 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:56,452 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:56,452 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,452 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:56,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:56,452 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,453 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:56,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,455 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,455 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,455 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,455 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,455 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,455 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:56,455 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:56,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:56,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:56,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:56,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:56,455 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:56,455 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,456 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:56,456 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:56,456 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,456 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:56,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:56,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:56,466 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:56,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53172 deadline: 1680922556464, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,556 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:56,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:56,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:56,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:56,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:56,560 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:56,560 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:56,560 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:56,560 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:56,560 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,560 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,562 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:56,562 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:56,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,563 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,563 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:56,563 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:56,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,563 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:56,564 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:56,564 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:56,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:56,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:56,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:56,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:56,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:56,564 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:56,564 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:56,564 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:56,564 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:56,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:56,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:56,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:56,565 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:56,566 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,566 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:56,566 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,566 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:56,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:56,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:56,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:56,568 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6f1338f0[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:56,568 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:56,568 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,569 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,569 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:56,569 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:56,569 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:56,569 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,569 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:56,570 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,570 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:56,570 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:56,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:56,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:56,570 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:56,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:56,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:56,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:56,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:56,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:56,573 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:56,573 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:56,573 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:56,573 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:56,573 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:56,578 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,578 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:56,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,578 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:56,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:56,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:56,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:56,578 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:56,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:56,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,579 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:56,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,582 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:56,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:56,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:56,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:56,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:56,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:56,582 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:56,582 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:56,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:56,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:56,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:56,669 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 199 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53172 deadline: 1680922556667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,682 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:56,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:56,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:56,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:56,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:56,686 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:56,686 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:56,686 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:56,686 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:56,687 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,687 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,688 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:56,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:56,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:56,688 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,688 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,688 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:56,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:56,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,689 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:56,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:56,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:56,690 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:56,690 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:56,690 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:56,690 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:56,690 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:56,690 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=334 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/34aa33d84ba649b18a9991a8c37937c1 2023-04-08 02:54:56,690 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:56,691 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:56,691 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:56,691 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:56,691 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:56,691 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:56,691 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,691 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:56,691 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:56,692 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,692 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:56,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:56,693 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,693 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:56,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:56,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:56,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:56,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:56,696 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d720209[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:56,696 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:56,696 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,697 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/10c38b75155a45dba721e52823c9e635 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/10c38b75155a45dba721e52823c9e635 2023-04-08 02:54:56,698 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,699 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:56,699 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:56,699 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:56,699 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,699 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:56,700 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:56,700 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:56,700 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:56,700 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:56,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:56,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:56,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:56,701 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/10c38b75155a45dba721e52823c9e635, entries=150, sequenceid=334, filesize=11.9 K 2023-04-08 02:54:56,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:56,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,702 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/98cd3f7a91d74f1a827f621c9712504e as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/98cd3f7a91d74f1a827f621c9712504e 2023-04-08 02:54:56,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:56,704 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:56,704 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:56,704 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:56,704 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:56,704 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:56,704 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:56,705 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,705 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,705 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,705 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:56,706 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:56,706 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:56,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,708 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:56,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:56,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,712 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/98cd3f7a91d74f1a827f621c9712504e, entries=150, sequenceid=334, filesize=11.9 K 2023-04-08 02:54:56,713 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/34aa33d84ba649b18a9991a8c37937c1 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/34aa33d84ba649b18a9991a8c37937c1 2023-04-08 02:54:56,713 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:56,713 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:56,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:56,713 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,714 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,713 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:56,714 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,714 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,714 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:56,714 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,714 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:56,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:56,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:56,714 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:56,714 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:56,714 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:56,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:56,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:56,718 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/34aa33d84ba649b18a9991a8c37937c1, entries=150, sequenceid=334, filesize=11.9 K 2023-04-08 02:54:56,719 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for d3e62a2534068dd21a6b490550cac850 in 464ms, sequenceid=334, compaction requested=true 2023-04-08 02:54:56,719 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:56,719 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:56,719 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:54:56,719 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:56,719 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:54:56,719 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:56,720 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:54:56,720 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/A is initiating minor compaction (all files) 2023-04-08 02:54:56,720 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/A in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:56,720 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/081f5868ffb9469b96ce4c11b3ad27a6, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/4a8ca4d0a9c5476d920418f9418b4f8e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/10c38b75155a45dba721e52823c9e635] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=36.4 K 2023-04-08 02:54:56,720 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:54:56,720 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/B is initiating minor compaction (all files) 2023-04-08 02:54:56,721 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/B in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:56,721 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 081f5868ffb9469b96ce4c11b3ad27a6, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1680922495060 2023-04-08 02:54:56,721 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/ad50f33edc224fca94917315d4681bb1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/2500fd73318b401dab88b03f7ef73e1d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/98cd3f7a91d74f1a827f621c9712504e] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=36.4 K 2023-04-08 02:54:56,721 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 4a8ca4d0a9c5476d920418f9418b4f8e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1680922495118 2023-04-08 02:54:56,721 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting ad50f33edc224fca94917315d4681bb1, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1680922495060 2023-04-08 02:54:56,721 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 10c38b75155a45dba721e52823c9e635, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1680922496245 2023-04-08 02:54:56,722 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 2500fd73318b401dab88b03f7ef73e1d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1680922495118 2023-04-08 02:54:56,722 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 98cd3f7a91d74f1a827f621c9712504e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1680922496245 2023-04-08 02:54:56,736 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#B#compaction#281 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:56,738 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#A#compaction#282 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:56,790 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/83223b976685460398248f86e8fd9d63 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/83223b976685460398248f86e8fd9d63 2023-04-08 02:54:56,796 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/A of d3e62a2534068dd21a6b490550cac850 into 83223b976685460398248f86e8fd9d63(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:56,796 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:56,796 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/A, priority=13, startTime=1680922496719; duration=0sec 2023-04-08 02:54:56,796 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:56,796 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:54:56,797 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37274 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:54:56,798 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/C is initiating minor compaction (all files) 2023-04-08 02:54:56,798 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/C in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:56,798 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/2852ab4ca852432ebb55d93e507b02cc, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/f635ca9c9f7e4f488eda116355d196ad, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/34aa33d84ba649b18a9991a8c37937c1] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=36.4 K 2023-04-08 02:54:56,798 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 2852ab4ca852432ebb55d93e507b02cc, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=303, earliestPutTs=1680922495060 2023-04-08 02:54:56,799 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting f635ca9c9f7e4f488eda116355d196ad, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=323, earliestPutTs=1680922495118 2023-04-08 02:54:56,799 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 34aa33d84ba649b18a9991a8c37937c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1680922496245 2023-04-08 02:54:56,806 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#C#compaction#283 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:56,815 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:56,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:56,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:56,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:56,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:56,821 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:56,821 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:56,821 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:56,821 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:56,821 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,822 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,823 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,823 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:56,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:56,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:56,823 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:56,823 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:56,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,824 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:56,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:56,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:56,824 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:56,825 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/1e0c45baffc74543b9373ea0a6481e57 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/1e0c45baffc74543b9373ea0a6481e57 2023-04-08 02:54:56,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:56,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:56,828 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:56,828 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:56,828 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:56,828 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:56,828 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:56,828 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:56,828 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:56,829 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-08 02:54:56,829 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:54:56,829 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:54:56,829 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:54:56,829 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:54:56,829 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:54:56,829 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:54:56,830 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/C of d3e62a2534068dd21a6b490550cac850 into 1e0c45baffc74543b9373ea0a6481e57(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:56,830 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:56,830 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/C, priority=13, startTime=1680922496719; duration=0sec 2023-04-08 02:54:56,831 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:56,846 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/871b98b8495048108c0e86861b1a49fa 2023-04-08 02:54:56,858 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/a42a4c94299d4aa2964634fbb1bb42dc 2023-04-08 02:54:56,973 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:56,973 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:56,992 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:56,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53172 deadline: 1680922556990, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,094 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:57,094 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53172 deadline: 1680922557093, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,187 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/d2392b9f910d4d619c052a392506e837 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d2392b9f910d4d619c052a392506e837 2023-04-08 02:54:57,192 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/B of d3e62a2534068dd21a6b490550cac850 into d2392b9f910d4d619c052a392506e837(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:57,192 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:57,192 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/B, priority=13, startTime=1680922496719; duration=0sec 2023-04-08 02:54:57,192 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:57,270 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=373 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/78b5b3fc994b496fa57ee4a9de040e19 2023-04-08 02:54:57,274 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/871b98b8495048108c0e86861b1a49fa as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/871b98b8495048108c0e86861b1a49fa 2023-04-08 02:54:57,278 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/871b98b8495048108c0e86861b1a49fa, entries=150, sequenceid=373, filesize=11.9 K 2023-04-08 02:54:57,280 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/a42a4c94299d4aa2964634fbb1bb42dc as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/a42a4c94299d4aa2964634fbb1bb42dc 2023-04-08 02:54:57,284 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/a42a4c94299d4aa2964634fbb1bb42dc, entries=150, sequenceid=373, filesize=11.9 K 2023-04-08 02:54:57,285 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/78b5b3fc994b496fa57ee4a9de040e19 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/78b5b3fc994b496fa57ee4a9de040e19 2023-04-08 02:54:57,289 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/78b5b3fc994b496fa57ee4a9de040e19, entries=150, sequenceid=373, filesize=11.9 K 2023-04-08 02:54:57,290 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for d3e62a2534068dd21a6b490550cac850 in 461ms, sequenceid=373, compaction requested=false 2023-04-08 02:54:57,290 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:57,290 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:57,290 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:57,290 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:57,290 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,290 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:57,290 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:57,293 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,293 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:57,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:57,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:57,294 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,294 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:57,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:57,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:57,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:57,296 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7a3274ee[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:57,296 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:57,296 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,297 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:57,297 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,297 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-08 02:54:57,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,297 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:57,297 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:57,297 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:54:57,297 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:57,298 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:57,297 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,298 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:57,298 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:54:57,298 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:57,298 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:54:57,298 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:57,299 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:57,299 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:57,299 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:57,299 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:57,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:57,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,309 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/e42beb478658418aa0f86abbaca66f75 2023-04-08 02:54:57,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:57,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:57,312 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:57,312 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:57,312 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:57,312 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:57,312 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:57,314 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,314 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:57,314 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,315 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:57,315 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:57,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,317 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:57,321 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,321 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:57,321 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:57,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:57,321 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:57,321 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:57,321 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,321 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,321 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,321 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,321 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:57,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:57,321 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:57,321 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:57,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:57,321 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:57,322 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:57,322 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:57,322 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/e6584fd4292743ed8e8c8629585481e4 2023-04-08 02:54:57,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922497330, max timestamp=1680922497351Num uniques 859; 2023-04-08 02:54:57,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:54:57,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922497330, max timestamp=1680922497351Num uniques 859; 2023-04-08 02:54:57,353 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:54:57,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922497330, max timestamp=1680922497351Num uniques 859; 2023-04-08 02:54:57,354 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:54:57,356 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=386 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/4375c3dcfce64b33be21d1694fab29be 2023-04-08 02:54:57,361 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/e42beb478658418aa0f86abbaca66f75 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/e42beb478658418aa0f86abbaca66f75 2023-04-08 02:54:57,365 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/e42beb478658418aa0f86abbaca66f75, entries=150, sequenceid=386, filesize=11.9 K 2023-04-08 02:54:57,367 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/e6584fd4292743ed8e8c8629585481e4 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e6584fd4292743ed8e8c8629585481e4 2023-04-08 02:54:57,371 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e6584fd4292743ed8e8c8629585481e4, entries=150, sequenceid=386, filesize=11.9 K 2023-04-08 02:54:57,371 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/4375c3dcfce64b33be21d1694fab29be as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/4375c3dcfce64b33be21d1694fab29be 2023-04-08 02:54:57,375 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/4375c3dcfce64b33be21d1694fab29be, entries=150, sequenceid=386, filesize=11.9 K 2023-04-08 02:54:57,383 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=67.09 KB/68700 for d3e62a2534068dd21a6b490550cac850 in 86ms, sequenceid=386, compaction requested=true 2023-04-08 02:54:57,383 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:57,383 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:57,383 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:57,383 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:54:57,383 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:57,383 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:54:57,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:57,383 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=73.80 KB heapSize=205.69 KB 2023-04-08 02:54:57,384 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:54:57,384 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:54:57,384 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:54:57,384 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:54:57,384 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:54:57,384 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:54:57,384 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:54:57,384 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:54:57,385 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/B is initiating minor compaction (all files) 2023-04-08 02:54:57,385 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/A is initiating minor compaction (all files) 2023-04-08 02:54:57,385 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/B in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:57,385 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/A in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:57,385 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d2392b9f910d4d619c052a392506e837, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/a42a4c94299d4aa2964634fbb1bb42dc, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e6584fd4292743ed8e8c8629585481e4] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=36.5 K 2023-04-08 02:54:57,385 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/83223b976685460398248f86e8fd9d63, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/871b98b8495048108c0e86861b1a49fa, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/e42beb478658418aa0f86abbaca66f75] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=36.5 K 2023-04-08 02:54:57,385 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting d2392b9f910d4d619c052a392506e837, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1680922496245 2023-04-08 02:54:57,385 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 83223b976685460398248f86e8fd9d63, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1680922496245 2023-04-08 02:54:57,385 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting a42a4c94299d4aa2964634fbb1bb42dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1680922496325 2023-04-08 02:54:57,386 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 871b98b8495048108c0e86861b1a49fa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1680922496325 2023-04-08 02:54:57,386 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting e6584fd4292743ed8e8c8629585481e4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1680922496986 2023-04-08 02:54:57,386 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting e42beb478658418aa0f86abbaca66f75, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1680922496986 2023-04-08 02:54:57,395 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#B#compaction#291 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:57,401 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#A#compaction#292 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:57,408 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=414 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/e270e87b4e454108804e9fd17003b1ac 2023-04-08 02:54:57,417 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/6e5a84ecf2e6404d92b5ea40501e6813 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/6e5a84ecf2e6404d92b5ea40501e6813 2023-04-08 02:54:57,422 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:57,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:57,425 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/B of d3e62a2534068dd21a6b490550cac850 into 6e5a84ecf2e6404d92b5ea40501e6813(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:57,425 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:57,425 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/B, priority=13, startTime=1680922497383; duration=0sec 2023-04-08 02:54:57,425 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:57,425 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:57,425 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:54:57,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:57,426 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37376 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:54:57,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:57,426 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/C is initiating minor compaction (all files) 2023-04-08 02:54:57,426 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:57,426 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/C in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:57,426 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:57,426 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/1e0c45baffc74543b9373ea0a6481e57, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/78b5b3fc994b496fa57ee4a9de040e19, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/4375c3dcfce64b33be21d1694fab29be] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=36.5 K 2023-04-08 02:54:57,426 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:57,426 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:57,427 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 1e0c45baffc74543b9373ea0a6481e57, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=334, earliestPutTs=1680922496245 2023-04-08 02:54:57,427 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,427 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:57,427 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=414 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/f52fd04a4e9f450481473ffb277b384f 2023-04-08 02:54:57,427 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 78b5b3fc994b496fa57ee4a9de040e19, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=373, earliestPutTs=1680922496325 2023-04-08 02:54:57,428 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 4375c3dcfce64b33be21d1694fab29be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1680922496986 2023-04-08 02:54:57,428 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,428 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:57,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:57,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:57,429 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,429 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:57,429 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:57,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:57,430 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:57,430 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:57,430 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:57,430 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:57,430 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:57,430 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:57,431 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:57,431 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:57,431 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:57,431 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:57,431 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:57,431 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:57,431 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:57,431 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:57,431 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,431 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:57,431 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:57,433 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,433 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:57,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:57,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,435 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,435 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:57,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:57,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:57,436 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3233db22[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:57,436 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:57,437 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,438 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,438 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:57,438 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:57,438 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:57,440 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,440 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:57,440 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:57,440 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:57,440 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:57,440 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:57,440 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:57,440 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:57,441 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:57,441 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:57,441 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:57,442 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#C#compaction#294 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:57,443 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,443 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:57,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,443 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:57,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:57,444 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:57,444 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:57,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,447 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:57,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,452 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:57,452 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:57,452 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:57,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:57,452 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:57,452 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:57,452 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:57,452 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:57,452 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:57,453 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,453 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,453 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,453 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,453 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:57,453 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:57,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:57,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:57,453 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:57,457 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:57,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 254 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53172 deadline: 1680922557454, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,469 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=414 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/6045515eb7e346e6ae7336b8ff5d34ea 2023-04-08 02:54:57,474 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/e270e87b4e454108804e9fd17003b1ac as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/e270e87b4e454108804e9fd17003b1ac 2023-04-08 02:54:57,480 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/e270e87b4e454108804e9fd17003b1ac, entries=150, sequenceid=414, filesize=11.9 K 2023-04-08 02:54:57,481 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/f52fd04a4e9f450481473ffb277b384f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/f52fd04a4e9f450481473ffb277b384f 2023-04-08 02:54:57,485 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/f52fd04a4e9f450481473ffb277b384f, entries=150, sequenceid=414, filesize=11.9 K 2023-04-08 02:54:57,486 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/6045515eb7e346e6ae7336b8ff5d34ea as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/6045515eb7e346e6ae7336b8ff5d34ea 2023-04-08 02:54:57,489 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/6045515eb7e346e6ae7336b8ff5d34ea, entries=150, sequenceid=414, filesize=11.9 K 2023-04-08 02:54:57,490 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~205.64 KB/210576, currentSize=107.34 KB/109920 for d3e62a2534068dd21a6b490550cac850 in 107ms, sequenceid=414, compaction requested=false 2023-04-08 02:54:57,490 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:57,553 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:57,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:57,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:57,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:57,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:57,556 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:57,556 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:57,556 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:57,556 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:57,557 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,557 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:57,558 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,558 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:57,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:57,558 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,558 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:57,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:57,559 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:57,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:57,559 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:57,559 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:57,559 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:57,559 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:57,559 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:57,560 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:57,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:57,560 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:57,560 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-08 02:54:57,560 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:57,560 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:57,560 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:57,560 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:57,560 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:54:57,560 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:57,560 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:57,560 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:57,560 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,560 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:57,560 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:57,560 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:54:57,560 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:57,560 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:57,560 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:54:57,560 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:57,567 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:57,567 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:57,567 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,567 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:57,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:57,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:57,569 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:57,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4491b12c[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:57,569 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,570 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,570 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:57,570 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:57,570 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:57,570 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,570 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:57,571 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:57,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:57,571 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:57,572 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:57,572 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:57,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,573 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=435 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/58cae7fbddb94205a0ec7368d5278df6 2023-04-08 02:54:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:57,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:57,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:57,574 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:57,574 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:57,574 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:57,574 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:57,574 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:57,576 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,576 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:57,576 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:57,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:57,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:57,577 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,578 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:57,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:57,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:57,582 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:57,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:57,582 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:57,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:57,582 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:57,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:57,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:57,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,583 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:57,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:57,583 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:57,583 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:57,591 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:57,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 266 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53172 deadline: 1680922557589, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,683 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:57,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:57,685 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:57,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:57,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:57,686 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:57,686 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:57,686 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:57,686 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:57,687 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,687 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:57,690 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,690 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:57,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:57,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:57,691 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,691 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:57,691 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:57,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:57,691 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:57,692 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:57,692 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:57,692 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:57,692 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:57,692 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:57,692 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:57,692 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:57,692 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:57,692 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:57,692 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:57,692 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:57,692 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:57,692 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:57,692 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,692 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:57,692 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:57,694 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,694 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:57,694 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:57,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 268 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53172 deadline: 1680922557692, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:57,694 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,694 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:57,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:57,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:57,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:57,695 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@19ae91fd[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:57,695 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:57,696 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,697 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,698 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,698 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:57,698 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:57,698 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:57,698 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:57,699 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,699 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:57,699 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:57,699 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:57,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:57,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:57,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:57,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:57,702 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:57,702 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:57,702 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:57,702 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:57,702 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:57,703 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,703 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,704 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:57,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:57,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:57,704 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:57,705 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:57,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,708 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:57,708 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:57,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:57,708 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:57,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:57,708 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:57,708 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:57,708 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:57,708 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:57,708 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,708 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,708 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,709 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:57,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:57,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:57,708 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:57,709 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:57,809 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:57,809 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:57,811 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:57,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:57,813 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:57,813 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:57,813 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:57,813 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:57,813 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:57,814 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/459f2ee807d74a08898ce44af69bdf28 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/459f2ee807d74a08898ce44af69bdf28 2023-04-08 02:54:57,814 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,814 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:57,816 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:57,816 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:57,816 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:57,816 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,816 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:57,817 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:57,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:57,817 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:57,818 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:57,818 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:57,818 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:57,818 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:57,818 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:57,818 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:57,818 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:57,818 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:57,818 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:57,819 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:57,819 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/A of d3e62a2534068dd21a6b490550cac850 into 459f2ee807d74a08898ce44af69bdf28(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:57,819 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:57,819 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:57,819 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:57,819 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/A, priority=13, startTime=1680922497383; duration=0sec 2023-04-08 02:54:57,819 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:57,819 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,819 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:57,819 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:57,819 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:57,820 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,820 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:57,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:57,821 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,821 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:57,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:57,822 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70520322[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:57,822 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:57,822 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,823 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,823 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,823 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:57,823 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:57,823 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:57,823 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,823 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:57,824 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,824 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:57,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:57,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:57,824 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:57,825 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:57,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:57,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:57,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:57,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:57,826 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:57,826 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:57,827 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:57,827 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:57,827 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:57,827 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:57,829 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,829 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:57,829 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:57,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:57,830 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:57,830 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:57,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,831 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,833 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,833 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:57,833 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:57,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:57,833 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:57,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:57,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:57,833 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:57,833 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:57,833 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:57,833 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:57,833 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,833 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,833 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:57,833 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,833 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:57,833 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,834 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:57,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:57,860 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/336d78fba99943299b5d757721c8b5ac as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/336d78fba99943299b5d757721c8b5ac 2023-04-08 02:54:57,865 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/C of d3e62a2534068dd21a6b490550cac850 into 336d78fba99943299b5d757721c8b5ac(size=12.8 K), total size for store is 24.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:57,865 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:57,865 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/C, priority=13, startTime=1680922497383; duration=0sec 2023-04-08 02:54:57,866 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:57,885 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:57,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53102 deadline: 1680922557884, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,887 DEBUG [Thread-1626] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4125 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:54:57,895 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:57,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1680922557894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,896 DEBUG [Thread-1624] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4133 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:54:57,898 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:57,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 270 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53172 deadline: 1680922557897, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:57,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922557900, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,902 DEBUG [Thread-1620] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4142 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:54:57,934 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:57,943 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:57,945 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:57,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:57,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:57,946 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:57,946 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:57,946 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:57,946 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:57,947 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,947 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:57,948 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:57,948 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:57,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:57,948 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,948 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:57,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:57,949 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:57,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:57,949 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:57,949 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:57,949 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:57,950 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:57,950 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:57,950 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:57,950 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:57,950 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:57,950 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:57,950 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:57,950 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:57,950 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:57,950 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:57,950 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,950 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:57,950 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:57,953 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,953 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:57,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:57,953 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,953 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:57,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:57,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:57,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:57,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:57,955 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:57,955 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5c142cc1[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:57,955 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,956 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,956 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:57,956 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:57,956 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:57,956 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,956 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:57,957 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,957 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:57,958 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:57,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,958 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:57,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:57,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:57,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:57,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:57,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:57,960 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:57,960 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:57,960 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:57,960 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:57,960 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:57,962 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,962 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,962 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:57,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:57,962 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:57,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,963 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:57,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,967 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,967 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:57,967 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:57,967 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:57,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:57,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:57,967 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:57,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:57,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:57,967 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:57,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:57,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:57,967 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:57,968 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:57,968 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:57,968 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,968 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:57,968 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:57,984 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=435 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/8e38f255f7ae486e8df976c3cf4993a8 2023-04-08 02:54:58,068 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:58,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:58,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:58,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:58,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:58,072 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:58,072 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:58,072 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:58,072 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:58,072 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,073 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:58,075 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:58,075 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:58,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:58,076 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,076 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:58,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:58,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:58,076 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:58,077 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:58,077 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:58,077 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:58,077 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:58,077 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:58,080 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:58,080 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:58,080 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:58,080 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:58,080 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:58,080 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:58,080 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:58,080 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:58,080 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,080 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:58,080 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:58,099 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:58,099 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:58,099 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:58,099 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:58,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:58,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:58,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:58,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:58,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:58,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@15c5febe[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:58,102 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:58,102 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:58,103 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:58,103 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:58,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:58,103 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:58,103 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:58,103 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:58,103 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,103 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:58,104 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,105 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:58,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:58,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:58,105 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:58,105 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:58,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:58,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:58,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:58,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:58,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:58,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:58,109 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:58,109 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:58,109 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:58,109 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:58,109 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:58,111 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,111 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:58,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:58,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,111 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:58,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:58,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:58,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:58,112 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:58,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:58,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:58,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,113 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:58,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:58,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,117 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,117 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:58,117 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:58,117 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,117 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:58,117 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:58,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:58,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:58,117 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:58,117 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:58,117 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,117 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:58,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:58,117 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:58,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:58,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:58,118 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:58,118 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:58,201 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:58,201 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 272 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53172 deadline: 1680922558200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,218 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:58,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:58,220 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:58,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:58,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:58,221 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:58,222 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:58,222 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:58,222 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:58,222 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,222 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:58,223 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:58,223 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:58,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:58,223 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,224 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:58,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:58,224 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:58,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:58,224 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:58,224 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:58,224 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:58,225 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:58,225 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:58,225 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:58,225 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:58,225 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:58,225 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:58,225 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:58,225 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:58,225 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:58,225 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:58,225 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:58,225 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,225 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:58,225 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:58,227 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:58,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:58,228 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:58,228 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:58,228 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:58,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:58,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:58,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7889a482[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:58,229 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:58,229 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:58,230 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:58,230 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:58,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:58,230 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:58,231 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:58,231 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,231 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:58,231 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:58,233 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:58,233 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:58,233 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,233 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:58,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:58,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:58,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:58,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:58,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:58,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:58,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:58,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:58,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:58,235 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:58,235 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:58,235 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:58,235 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:58,235 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:58,236 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,237 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,237 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:58,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,237 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:58,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:58,238 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:58,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:58,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:58,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,240 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,240 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:58,240 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:58,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:58,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:58,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:58,240 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:58,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:58,240 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:58,240 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:58,240 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:58,240 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:58,240 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:58,240 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:58,241 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:58,241 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:58,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:58,340 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:58,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:58,343 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:58,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:58,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:58,344 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:58,344 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:58,344 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:58,344 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:58,344 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,344 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:58,345 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,345 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:58,346 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:58,346 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,346 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:58,346 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:58,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:58,347 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:58,347 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:58,347 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:58,347 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:58,347 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:58,347 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:58,347 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:58,347 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:58,347 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:58,347 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:58,348 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:58,348 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:58,348 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:58,348 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:58,348 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,348 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:58,348 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:58,350 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,350 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:58,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:58,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:58,350 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:58,350 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:58,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:58,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:58,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:58,352 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1822f3a2[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:58,352 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:58,352 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:58,353 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:58,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:58,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:58,353 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,353 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:58,353 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:58,353 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:58,353 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:58,354 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,354 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:58,355 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:58,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:58,355 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:58,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:58,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:58,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:58,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:58,357 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:58,357 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:58,357 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:58,357 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:58,357 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:58,357 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:58,359 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,359 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:58,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,359 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:58,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:58,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:58,360 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:58,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:58,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:58,360 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:58,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:58,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:58,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:58,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,363 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,371 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,371 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:58,371 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:58,371 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:58,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:58,371 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:58,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:58,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:58,371 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:58,371 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:58,372 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,372 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:58,372 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:58,372 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,372 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:58,372 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:58,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:58,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:58,395 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=435 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/3a0b16f6d510451a8d1a150ff35de2c8 2023-04-08 02:54:58,398 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/58cae7fbddb94205a0ec7368d5278df6 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/58cae7fbddb94205a0ec7368d5278df6 2023-04-08 02:54:58,402 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/58cae7fbddb94205a0ec7368d5278df6, entries=150, sequenceid=435, filesize=11.9 K 2023-04-08 02:54:58,402 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/8e38f255f7ae486e8df976c3cf4993a8 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/8e38f255f7ae486e8df976c3cf4993a8 2023-04-08 02:54:58,407 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/8e38f255f7ae486e8df976c3cf4993a8, entries=150, sequenceid=435, filesize=11.9 K 2023-04-08 02:54:58,407 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/3a0b16f6d510451a8d1a150ff35de2c8 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/3a0b16f6d510451a8d1a150ff35de2c8 2023-04-08 02:54:58,411 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/3a0b16f6d510451a8d1a150ff35de2c8, entries=150, sequenceid=435, filesize=11.9 K 2023-04-08 02:54:58,412 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for d3e62a2534068dd21a6b490550cac850 in 852ms, sequenceid=435, compaction requested=true 2023-04-08 02:54:58,412 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:58,412 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:58,412 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:54:58,412 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:54:58,412 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:58,412 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:58,414 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:54:58,414 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:54:58,414 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/A is initiating minor compaction (all files) 2023-04-08 02:54:58,414 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/B is initiating minor compaction (all files) 2023-04-08 02:54:58,414 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/A in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:58,414 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/B in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:58,414 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/459f2ee807d74a08898ce44af69bdf28, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/e270e87b4e454108804e9fd17003b1ac, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/58cae7fbddb94205a0ec7368d5278df6] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=36.6 K 2023-04-08 02:54:58,414 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/6e5a84ecf2e6404d92b5ea40501e6813, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/f52fd04a4e9f450481473ffb277b384f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/8e38f255f7ae486e8df976c3cf4993a8] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=36.6 K 2023-04-08 02:54:58,415 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 459f2ee807d74a08898ce44af69bdf28, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1680922496986 2023-04-08 02:54:58,415 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 6e5a84ecf2e6404d92b5ea40501e6813, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1680922496986 2023-04-08 02:54:58,415 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting e270e87b4e454108804e9fd17003b1ac, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1680922497361 2023-04-08 02:54:58,415 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting f52fd04a4e9f450481473ffb277b384f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1680922497361 2023-04-08 02:54:58,415 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 58cae7fbddb94205a0ec7368d5278df6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1680922497448 2023-04-08 02:54:58,415 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 8e38f255f7ae486e8df976c3cf4993a8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1680922497448 2023-04-08 02:54:58,423 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#A#compaction#299 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:58,429 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#B#compaction#300 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:58,446 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/e0e09aa2d81b47ac8c9fed792c22ac8f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e0e09aa2d81b47ac8c9fed792c22ac8f 2023-04-08 02:54:58,449 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/2f1c303e29a14c18958c93e16a30b254 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/2f1c303e29a14c18958c93e16a30b254 2023-04-08 02:54:58,451 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/B of d3e62a2534068dd21a6b490550cac850 into e0e09aa2d81b47ac8c9fed792c22ac8f(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:58,451 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:58,451 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/B, priority=13, startTime=1680922498412; duration=0sec 2023-04-08 02:54:58,451 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:54:58,451 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:54:58,453 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:54:58,453 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/C is initiating minor compaction (all files) 2023-04-08 02:54:58,453 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/C in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:58,453 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/336d78fba99943299b5d757721c8b5ac, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/6045515eb7e346e6ae7336b8ff5d34ea, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/3a0b16f6d510451a8d1a150ff35de2c8] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=36.6 K 2023-04-08 02:54:58,453 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 336d78fba99943299b5d757721c8b5ac, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=386, earliestPutTs=1680922496986 2023-04-08 02:54:58,454 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 6045515eb7e346e6ae7336b8ff5d34ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=414, earliestPutTs=1680922497361 2023-04-08 02:54:58,454 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 3a0b16f6d510451a8d1a150ff35de2c8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1680922497448 2023-04-08 02:54:58,454 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/A of d3e62a2534068dd21a6b490550cac850 into 2f1c303e29a14c18958c93e16a30b254(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:58,455 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:58,455 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/A, priority=13, startTime=1680922498412; duration=0sec 2023-04-08 02:54:58,455 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:58,460 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#C#compaction#301 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:54:58,470 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/8fb05f56362744b1af3ef8f1d14e150d as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/8fb05f56362744b1af3ef8f1d14e150d 2023-04-08 02:54:58,472 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:58,472 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:58,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:58,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:58,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:58,475 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:58,475 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:58,475 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:58,475 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:58,475 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/C of d3e62a2534068dd21a6b490550cac850 into 8fb05f56362744b1af3ef8f1d14e150d(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:54:58,476 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:58,476 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/C, priority=13, startTime=1680922498412; duration=0sec 2023-04-08 02:54:58,476 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,476 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:58,476 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:54:58,477 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:58,477 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:58,477 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,477 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:58,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:58,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:58,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:58,478 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:58,478 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:58,478 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:58,478 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:58,478 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:58,479 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:58,480 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:58,480 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:58,480 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:58,480 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:58,480 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-08 02:54:58,480 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:54:58,480 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:58,480 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:54:58,480 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:58,480 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:54:58,480 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:58,490 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:58,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:58,491 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/77606c9ad16e46a4b28652b408ebe5bc 2023-04-08 02:54:58,543 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:58,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922558542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,645 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:58,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922558644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,705 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:58,705 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 274 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53172 deadline: 1680922558704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,848 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:58,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922558847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:58,918 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/d614477ba87b4837855e8c38f336576e 2023-04-08 02:54:59,151 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:59,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 100 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922559151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:59,329 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=452 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/11d926fd80854bcf8bd50c9419ad032d 2023-04-08 02:54:59,334 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/77606c9ad16e46a4b28652b408ebe5bc as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/77606c9ad16e46a4b28652b408ebe5bc 2023-04-08 02:54:59,338 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/77606c9ad16e46a4b28652b408ebe5bc, entries=150, sequenceid=452, filesize=11.9 K 2023-04-08 02:54:59,338 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/d614477ba87b4837855e8c38f336576e as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d614477ba87b4837855e8c38f336576e 2023-04-08 02:54:59,342 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d614477ba87b4837855e8c38f336576e, entries=150, sequenceid=452, filesize=11.9 K 2023-04-08 02:54:59,343 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/11d926fd80854bcf8bd50c9419ad032d as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/11d926fd80854bcf8bd50c9419ad032d 2023-04-08 02:54:59,346 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/11d926fd80854bcf8bd50c9419ad032d, entries=150, sequenceid=452, filesize=11.9 K 2023-04-08 02:54:59,346 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for d3e62a2534068dd21a6b490550cac850 in 866ms, sequenceid=452, compaction requested=false 2023-04-08 02:54:59,347 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:54:59,347 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:59,347 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:54:59,347 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:54:59,347 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:59,348 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:54:59,348 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:54:59,349 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:59,349 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:59,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:59,350 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:59,350 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:54:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:59,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:59,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:59,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:59,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:54:59,351 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@652db643[Count = 0] remaining members to acquire global barrier 2023-04-08 02:54:59,351 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:54:59,351 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:59,352 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:59,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:59,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:59,352 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:54:59,352 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:54:59,352 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:59,353 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:54:59,353 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:54:59,357 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:59,357 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:54:59,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:59,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:59,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:59,357 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:59,357 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:54:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:59,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:59,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:59,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:59,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:59,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:54:59,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:54:59,361 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:54:59,361 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:54:59,361 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:54:59,361 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:54:59,361 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:54:59,362 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:59,362 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:59,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:59,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:59,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:59,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:54:59,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:54:59,363 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:59,363 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:59,363 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:59,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:59,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:54:59,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:59,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:59,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:54:59,364 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:59,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:54:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:54:59,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:59,366 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:59,366 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:59,366 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:59,366 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:59,366 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:54:59,366 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:59,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:59,366 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:54:59,367 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:54:59,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:54:59,367 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:54:59,366 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:59,367 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:59,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:54:59,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:54:59,367 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:54:59,367 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:59,367 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:54:59,367 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:54:59,467 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:54:59,468 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:54:59,469 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:54:59,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:54:59,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:54:59,471 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:54:59,471 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:59,471 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:54:59,471 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:54:59,471 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:59,471 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:59,472 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:54:59,472 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:59,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:54:59,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:54:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:59,473 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:59,473 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:54:59,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:54:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:54:59,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:54:59,473 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:54:59,473 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:54:59,473 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:54:59,474 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:54:59,474 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:54:59,474 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:54:59,474 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:54:59,474 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:54:59,474 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:54:59,474 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:54:59,474 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-08 02:54:59,474 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:54:59,474 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:59,474 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:54:59,474 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:59,474 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:54:59,474 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:54:59,656 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:54:59,657 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:54:59,679 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:59,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922559677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:59,712 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:59,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 276 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53172 deadline: 1680922559711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:59,781 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:59,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922559780, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:54:59,882 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=472 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/69db4680af2c4b799dfaf2656aacbea1 2023-04-08 02:54:59,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:54:59,985 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922559983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,288 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:00,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53126 deadline: 1680922560288, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,292 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=472 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/c8e40f4bfbd74e23b8c86301d1a28792 2023-04-08 02:55:00,703 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=472 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/c0b5e09ab75544c2819c463cabc76a75 2023-04-08 02:55:00,707 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/69db4680af2c4b799dfaf2656aacbea1 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/69db4680af2c4b799dfaf2656aacbea1 2023-04-08 02:55:00,711 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/69db4680af2c4b799dfaf2656aacbea1, entries=150, sequenceid=472, filesize=11.9 K 2023-04-08 02:55:00,711 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/c8e40f4bfbd74e23b8c86301d1a28792 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/c8e40f4bfbd74e23b8c86301d1a28792 2023-04-08 02:55:00,715 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/c8e40f4bfbd74e23b8c86301d1a28792, entries=150, sequenceid=472, filesize=11.9 K 2023-04-08 02:55:00,716 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/c0b5e09ab75544c2819c463cabc76a75 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/c0b5e09ab75544c2819c463cabc76a75 2023-04-08 02:55:00,719 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/c0b5e09ab75544c2819c463cabc76a75, entries=150, sequenceid=472, filesize=11.9 K 2023-04-08 02:55:00,720 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for d3e62a2534068dd21a6b490550cac850 in 1246ms, sequenceid=472, compaction requested=true 2023-04-08 02:55:00,720 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:00,720 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:00,720 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:00,720 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:00,720 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,720 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:00,720 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:00,721 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,721 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:00,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:00,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:00,722 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:00,722 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:00,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:00,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:00,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:00,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:00,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:00,723 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f279456[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:00,723 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:00,723 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:00,725 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:00,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:00,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:00,725 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:00,725 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:00,725 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,725 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:00,725 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:00,727 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,727 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:00,728 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:00,728 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:00,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:00,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:00,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:00,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:00,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:00,730 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:00,730 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:00,730 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:00,730 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:00,730 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:00,730 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:00,732 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:00,732 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:00,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:00,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:00,732 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:00,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:00,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:00,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:00,732 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:00,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:00,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:00,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:00,733 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:00,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:00,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:00,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,734 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:00,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:00,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,736 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,736 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:00,736 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:00,736 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:00,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:00,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:00,736 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:00,736 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,736 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:00,736 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:00,736 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:00,736 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:00,736 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:00,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:00,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:00,736 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:00,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:00,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:00,795 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:55:00,795 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-08 02:55:00,795 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:55:00,795 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:00,795 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:55:00,795 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:00,796 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:55:00,796 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:00,808 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/7123a3c634794da2a20b38c6754c212a 2023-04-08 02:55:00,837 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:00,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:00,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:00,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:00,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:00,840 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:00,840 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:00,840 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:00,840 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:00,840 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:00,840 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:00,841 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:00,841 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:00,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:00,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:00,842 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,842 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:00,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:00,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:00,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:00,843 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:00,843 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:00,843 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:00,843 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:00,843 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:00,843 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:00,843 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:00,843 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:00,843 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:55:00,843 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:55:00,843 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:00,844 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:00,844 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:00,844 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,844 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:00,844 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:00,846 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:00,846 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:00,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:00,847 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:00,847 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:00,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:00,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:00,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:00,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:00,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:00,848 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6abf9684[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:00,848 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:00,848 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:00,849 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:00,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:00,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:00,849 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:00,849 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:00,849 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:00,849 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,849 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:00,851 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:00,851 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,851 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:00,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:00,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:00,851 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:00,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:00,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:00,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:00,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:00,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:00,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:00,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:00,853 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:00,853 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:00,853 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:00,853 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:00,853 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:00,855 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:00,855 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:00,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:00,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:00,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:00,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:00,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:00,855 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:00,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:00,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:00,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:00,855 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:00,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:00,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:00,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:00,856 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:00,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:00,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,860 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,860 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:00,860 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:00,860 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:00,860 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:00,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:00,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:00,860 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:00,860 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:00,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:00,860 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:00,860 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:00,860 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:00,860 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:00,860 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:00,860 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:00,860 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:00,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:00,863 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:00,863 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53126 deadline: 1680922560861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,960 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:00,961 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:00,962 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:00,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:00,963 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:00,963 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:00,964 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:00,964 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:00,964 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:00,964 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:00,964 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:00,965 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:00,965 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53126 deadline: 1680922560964, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,965 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,965 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:00,965 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:00,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:00,966 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,966 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:00,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:00,966 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:00,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:00,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:00,966 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:00,967 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:00,967 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:00,967 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:00,967 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:00,967 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:00,967 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:00,967 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:00,967 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:00,967 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:55:00,967 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:55:00,967 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:00,967 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:00,967 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:00,967 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,967 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:00,967 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:00,969 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:00,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:00,969 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:00,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:00,969 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:00,969 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:00,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:00,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:00,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:00,970 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:00,971 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@658fd792[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:00,971 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:00,971 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:00,972 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:00,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:00,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:00,972 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:00,972 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,972 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:00,972 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:00,972 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:00,974 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,974 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:00,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:00,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:00,974 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:00,975 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:00,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:00,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:00,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:00,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:00,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:00,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:00,977 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:00,977 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:00,977 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:00,977 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:00,977 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:00,977 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:00,981 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:00,981 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:00,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:00,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:00,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:00,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:00,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:00,981 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:00,982 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:00,982 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:00,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:00,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:00,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,983 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,983 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:00,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:00,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,985 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,985 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:00,985 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:00,985 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:00,985 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:00,985 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:00,985 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:00,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:00,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:00,986 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:00,986 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:00,989 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:00,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:00,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:00,990 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:00,990 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:00,990 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:00,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:01,090 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:01,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:01,092 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:01,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:01,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:01,093 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:01,093 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:01,093 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:01,093 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:01,094 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:01,094 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:01,096 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,096 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:01,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:01,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:01,096 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,096 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:01,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:01,096 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:01,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:01,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:01,097 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:01,097 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:01,097 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:01,097 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:01,097 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:01,097 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:01,097 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:01,097 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:01,097 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:01,097 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:55:01,097 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:55:01,097 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:01,097 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:01,098 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:01,098 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,098 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:01,098 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:01,099 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,099 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:01,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:01,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:01,099 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:01,099 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:01,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:01,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:01,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:01,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:01,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:01,101 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6d98eace[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:01,101 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:01,101 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:01,102 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:01,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:01,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:01,102 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:01,102 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:01,102 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:01,102 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,102 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:01,103 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:01,103 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:01,103 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,103 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:01,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:01,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:01,106 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:01,106 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:01,106 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:01,106 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:01,106 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:01,106 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:01,109 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:01,109 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:01,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:01,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:01,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:01,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:01,109 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:01,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:01,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:01,109 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,109 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:01,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:01,110 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:01,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:01,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,113 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,113 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:01,113 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:01,113 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:01,113 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:01,113 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:01,113 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:01,113 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:01,113 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:01,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:01,113 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:01,113 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:01,113 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:01,113 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:01,113 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:01,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:01,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:01,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:01,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53126 deadline: 1680922561166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,213 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:01,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:01,215 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:01,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:01,216 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:01,217 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:01,217 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:01,217 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:01,217 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:01,217 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/600c035254634338838bf4601ba3bd28 2023-04-08 02:55:01,217 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:01,217 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:01,218 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:01,218 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,219 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:01,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:01,219 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,219 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:01,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:01,219 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:01,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:01,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:01,220 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:01,220 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:01,220 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:01,220 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:01,220 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:01,220 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:01,221 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:01,221 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:01,221 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:01,221 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:55:01,221 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:55:01,221 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:01,221 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:01,221 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:01,221 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,221 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:01,221 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:01,223 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,223 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:01,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:01,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:01,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:01,224 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:01,224 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:01,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:01,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:01,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:01,225 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:01,226 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@efe560f[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:01,226 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:01,226 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:01,228 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:01,228 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:01,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:01,228 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:01,228 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:01,228 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:01,228 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,228 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:01,229 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=485 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/e7fc26e2c2d04f1bbd47893e82d943aa 2023-04-08 02:55:01,230 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,230 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:01,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:01,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:01,230 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:01,230 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:01,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:01,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:01,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:01,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:01,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:01,233 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:01,233 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:01,233 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:01,233 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:01,233 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:01,233 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:01,234 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/7123a3c634794da2a20b38c6754c212a as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/7123a3c634794da2a20b38c6754c212a 2023-04-08 02:55:01,235 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:01,235 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:01,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:01,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:01,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:01,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:01,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:01,235 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:01,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:01,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:01,236 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:01,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:01,238 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,238 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/7123a3c634794da2a20b38c6754c212a, entries=200, sequenceid=485, filesize=14.3 K 2023-04-08 02:55:01,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:01,239 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/600c035254634338838bf4601ba3bd28 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/600c035254634338838bf4601ba3bd28 2023-04-08 02:55:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:01,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:01,241 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,241 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:01,241 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:01,241 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:01,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:01,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:01,241 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:01,241 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:01,241 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:01,242 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:01,241 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,242 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:01,242 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:01,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:01,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:01,242 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:01,242 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:01,242 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:01,242 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:01,243 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/600c035254634338838bf4601ba3bd28, entries=150, sequenceid=485, filesize=11.9 K 2023-04-08 02:55:01,243 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/e7fc26e2c2d04f1bbd47893e82d943aa as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/e7fc26e2c2d04f1bbd47893e82d943aa 2023-04-08 02:55:01,247 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/e7fc26e2c2d04f1bbd47893e82d943aa, entries=150, sequenceid=485, filesize=11.9 K 2023-04-08 02:55:01,248 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for d3e62a2534068dd21a6b490550cac850 in 453ms, sequenceid=485, compaction requested=true 2023-04-08 02:55:01,248 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:01,248 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:01,248 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:55:01,248 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:55:01,248 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:01,249 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:01,250 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49766 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:55:01,250 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/B is initiating minor compaction (all files) 2023-04-08 02:55:01,250 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/B in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:01,250 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e0e09aa2d81b47ac8c9fed792c22ac8f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d614477ba87b4837855e8c38f336576e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/c8e40f4bfbd74e23b8c86301d1a28792, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/600c035254634338838bf4601ba3bd28] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=48.6 K 2023-04-08 02:55:01,251 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52206 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:55:01,251 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/A is initiating minor compaction (all files) 2023-04-08 02:55:01,251 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/A in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:01,251 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/2f1c303e29a14c18958c93e16a30b254, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/77606c9ad16e46a4b28652b408ebe5bc, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/69db4680af2c4b799dfaf2656aacbea1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/7123a3c634794da2a20b38c6754c212a] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=51.0 K 2023-04-08 02:55:01,251 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting e0e09aa2d81b47ac8c9fed792c22ac8f, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1680922497448 2023-04-08 02:55:01,251 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 2f1c303e29a14c18958c93e16a30b254, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1680922497448 2023-04-08 02:55:01,252 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting d614477ba87b4837855e8c38f336576e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1680922497573 2023-04-08 02:55:01,252 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 77606c9ad16e46a4b28652b408ebe5bc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1680922497573 2023-04-08 02:55:01,252 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting c8e40f4bfbd74e23b8c86301d1a28792, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=472, earliestPutTs=1680922498536 2023-04-08 02:55:01,252 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 69db4680af2c4b799dfaf2656aacbea1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=472, earliestPutTs=1680922498536 2023-04-08 02:55:01,252 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 600c035254634338838bf4601ba3bd28, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1680922499659 2023-04-08 02:55:01,253 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 7123a3c634794da2a20b38c6754c212a, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1680922499659 2023-04-08 02:55:01,260 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#A#compaction#312 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:01,260 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#B#compaction#311 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:01,342 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:01,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:01,344 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:01,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:01,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:01,346 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:01,346 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:01,346 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:01,346 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:01,346 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:01,346 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:01,348 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,348 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:01,348 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:01,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:01,349 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:01,349 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:01,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:01,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:01,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:01,349 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:01,350 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:01,350 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:01,350 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:01,350 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:01,350 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:01,350 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:01,350 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:01,350 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:01,350 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:55:01,351 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-08 02:55:01,351 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:55:01,351 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:01,351 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:55:01,351 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:01,351 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:55:01,351 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:01,470 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:55:01,470 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:55:01,503 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:01,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53126 deadline: 1680922561500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,606 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:01,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53126 deadline: 1680922561604, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,670 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/65afa1e381f04a04ba887afd9f533488 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/65afa1e381f04a04ba887afd9f533488 2023-04-08 02:55:01,670 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/ae982203a8da425b8d9333063dc7ef8f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/ae982203a8da425b8d9333063dc7ef8f 2023-04-08 02:55:01,675 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d3e62a2534068dd21a6b490550cac850/B of d3e62a2534068dd21a6b490550cac850 into 65afa1e381f04a04ba887afd9f533488(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:01,675 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:01,675 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/B, priority=12, startTime=1680922501248; duration=0sec 2023-04-08 02:55:01,675 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:01,675 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:55:01,675 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d3e62a2534068dd21a6b490550cac850/A of d3e62a2534068dd21a6b490550cac850 into ae982203a8da425b8d9333063dc7ef8f(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:01,675 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:01,675 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/A, priority=12, startTime=1680922501248; duration=0sec 2023-04-08 02:55:01,675 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:01,676 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49766 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:55:01,676 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/C is initiating minor compaction (all files) 2023-04-08 02:55:01,676 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/C in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:01,676 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/8fb05f56362744b1af3ef8f1d14e150d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/11d926fd80854bcf8bd50c9419ad032d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/c0b5e09ab75544c2819c463cabc76a75, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/e7fc26e2c2d04f1bbd47893e82d943aa] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=48.6 K 2023-04-08 02:55:01,677 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 8fb05f56362744b1af3ef8f1d14e150d, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=435, earliestPutTs=1680922497448 2023-04-08 02:55:01,677 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 11d926fd80854bcf8bd50c9419ad032d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=452, earliestPutTs=1680922497573 2023-04-08 02:55:01,678 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting c0b5e09ab75544c2819c463cabc76a75, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=472, earliestPutTs=1680922498536 2023-04-08 02:55:01,678 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting e7fc26e2c2d04f1bbd47893e82d943aa, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1680922499659 2023-04-08 02:55:01,685 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#C#compaction#314 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:01,693 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/d12e7e6f606a4a70892bcc65bfdf1bbf as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/d12e7e6f606a4a70892bcc65bfdf1bbf 2023-04-08 02:55:01,697 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d3e62a2534068dd21a6b490550cac850/C of d3e62a2534068dd21a6b490550cac850 into d12e7e6f606a4a70892bcc65bfdf1bbf(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:01,697 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:01,697 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/C, priority=12, startTime=1680922501249; duration=0sec 2023-04-08 02:55:01,697 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:01,732 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:01,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 278 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53172 deadline: 1680922561731, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,733 DEBUG [Thread-1622] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:55:01,759 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/a4c20daf19284992b2f78e4224508266 2023-04-08 02:55:01,810 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:01,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53126 deadline: 1680922561808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,910 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:01,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53102 deadline: 1680922561909, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,912 DEBUG [Thread-1626] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8150 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:55:01,925 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:01,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53122 deadline: 1680922561925, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,927 DEBUG [Thread-1624] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8164 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:55:01,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:01,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:53178 deadline: 1680922561934, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:01,937 DEBUG [Thread-1620] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8177 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:55:02,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:02,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53126 deadline: 1680922562111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,168 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/ec5c11b9f14441d5ab5a9b71f185b4fc 2023-04-08 02:55:02,181 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=504 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/f7167b5985104042963f2f27701c8c71 2023-04-08 02:55:02,185 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/a4c20daf19284992b2f78e4224508266 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/a4c20daf19284992b2f78e4224508266 2023-04-08 02:55:02,189 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/a4c20daf19284992b2f78e4224508266, entries=150, sequenceid=504, filesize=11.9 K 2023-04-08 02:55:02,190 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/ec5c11b9f14441d5ab5a9b71f185b4fc as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/ec5c11b9f14441d5ab5a9b71f185b4fc 2023-04-08 02:55:02,213 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/ec5c11b9f14441d5ab5a9b71f185b4fc, entries=150, sequenceid=504, filesize=11.9 K 2023-04-08 02:55:02,214 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/f7167b5985104042963f2f27701c8c71 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/f7167b5985104042963f2f27701c8c71 2023-04-08 02:55:02,230 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/f7167b5985104042963f2f27701c8c71, entries=150, sequenceid=504, filesize=11.9 K 2023-04-08 02:55:02,230 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for d3e62a2534068dd21a6b490550cac850 in 879ms, sequenceid=504, compaction requested=false 2023-04-08 02:55:02,231 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:02,231 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:02,231 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:02,231 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:02,231 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,231 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:02,231 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:02,232 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,232 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:02,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:02,233 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,233 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:02,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:02,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:02,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:02,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:02,234 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@18e3eae2[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:02,234 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:02,234 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,235 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,235 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:02,235 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:02,235 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:02,236 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,236 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:02,237 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:02,237 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:02,237 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,237 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:02,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:02,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:02,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:02,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:02,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:02,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:02,239 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:02,239 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:02,239 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:02,239 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:02,239 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:02,239 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:02,244 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,244 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:02,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:02,244 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:02,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:02,244 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:02,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:02,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:02,245 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:02,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,250 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,250 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:02,250 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:02,250 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:02,250 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:02,250 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:02,250 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:02,250 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:02,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:02,250 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,250 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,250 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,250 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:02,250 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:02,250 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:02,250 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:02,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:02,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:02,351 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:02,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:02,353 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:02,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:02,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:02,354 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:02,354 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:02,354 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:02,354 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:02,355 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,355 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:02,358 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:02,358 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,358 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:02,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:02,358 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,359 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:02,359 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:02,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:02,359 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:02,360 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:02,360 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:02,360 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:02,360 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:02,360 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:02,360 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:02,360 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:02,360 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:02,360 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:55:02,361 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-08 02:55:02,361 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:55:02,361 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:02,361 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:55:02,361 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:02,361 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:55:02,361 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:02,372 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/5effe2bd68f54744a2c86cc12cbe380f 2023-04-08 02:55:02,390 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/39322bf082634a479465bdac26ec9df6 2023-04-08 02:55:02,413 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=520 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/651757c81ba94e7ab7554ea4a3688447 2023-04-08 02:55:02,420 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/5effe2bd68f54744a2c86cc12cbe380f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/5effe2bd68f54744a2c86cc12cbe380f 2023-04-08 02:55:02,425 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/5effe2bd68f54744a2c86cc12cbe380f, entries=150, sequenceid=520, filesize=11.9 K 2023-04-08 02:55:02,426 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/39322bf082634a479465bdac26ec9df6 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/39322bf082634a479465bdac26ec9df6 2023-04-08 02:55:02,431 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/39322bf082634a479465bdac26ec9df6, entries=150, sequenceid=520, filesize=11.9 K 2023-04-08 02:55:02,431 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/651757c81ba94e7ab7554ea4a3688447 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/651757c81ba94e7ab7554ea4a3688447 2023-04-08 02:55:02,436 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/651757c81ba94e7ab7554ea4a3688447, entries=150, sequenceid=520, filesize=11.9 K 2023-04-08 02:55:02,437 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=0 B/0 for d3e62a2534068dd21a6b490550cac850 in 77ms, sequenceid=520, compaction requested=true 2023-04-08 02:55:02,437 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:02,437 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:02,437 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:02,437 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:02,437 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,440 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:02,440 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:02,442 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,442 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:02,443 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,443 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:02,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:02,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:02,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:02,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:02,445 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cb9d2b[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:02,445 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:02,445 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,447 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,447 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:02,447 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:02,447 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:02,447 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,447 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:02,448 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:02,448 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,448 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:02,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:02,448 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:02,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:02,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:02,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:02,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:02,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:02,452 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:02,452 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:02,452 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:02,452 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:02,452 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:02,453 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,453 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,453 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,453 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:02,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:02,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:02,454 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:02,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:02,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:02,455 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:02,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:02,456 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,462 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:02,462 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:02,462 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,462 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:02,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:02,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:02,462 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:02,462 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:02,462 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:02,463 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:02,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:02,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:02,462 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:02,463 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:02,463 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,463 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,463 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,463 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,563 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:02,563 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:02,565 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:02,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:02,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:02,567 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:02,567 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:02,567 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:02,567 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:02,567 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,567 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:02,569 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,569 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:02,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:02,569 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,569 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:02,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:02,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:02,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:02,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:02,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:02,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:02,570 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:02,570 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:02,570 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:02,570 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:02,570 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:02,570 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:02,570 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:55:02,571 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:02,571 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:02,571 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:02,571 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:02,571 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,571 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:02,571 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:02,572 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:02,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:02,572 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:02,573 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,573 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:02,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:02,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:02,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@292fc906[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:02,574 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:02,575 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,576 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,576 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:02,576 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:02,576 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:02,576 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,576 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:02,578 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:02,578 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:02,578 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:02,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:02,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:02,578 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:02,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:02,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:02,582 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:02,582 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:02,582 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:02,582 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:02,582 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:02,582 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:02,584 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,584 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:02,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,584 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:02,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:02,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:02,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:02,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:02,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,585 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:02,587 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,592 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,592 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:02,592 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:02,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:02,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:02,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:02,592 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:02,592 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:02,592 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,592 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,592 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,592 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,592 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:02,592 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:02,592 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:02,592 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:02,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:02,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:02,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:55:02,637 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-08 02:55:02,638 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:55:02,638 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:02,638 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:55:02,638 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:02,638 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:55:02,638 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:02,654 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/ce9ac13585d84af1a17ac92f23862440 2023-04-08 02:55:02,668 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/3a39b47b08774b60874ff633656440f0 2023-04-08 02:55:02,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922502659, max timestamp=1680922502688Num uniques 859; 2023-04-08 02:55:02,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922502659, max timestamp=1680922502688Num uniques 859; 2023-04-08 02:55:02,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922502659, max timestamp=1680922502688Num uniques 859; 2023-04-08 02:55:02,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:55:02,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:55:02,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:55:02,692 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:02,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:02,694 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:02,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:02,695 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:02,695 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:02,695 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:02,695 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:02,695 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:02,696 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,696 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:02,697 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,697 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:02,697 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:02,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:02,697 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,697 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:02,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:02,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:02,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:02,698 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:02,698 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:02,698 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:02,699 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:02,699 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:02,699 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:02,699 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:02,699 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:02,699 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:02,699 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:55:02,699 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:55:02,699 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:02,699 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:02,699 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:02,699 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,699 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:02,699 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:02,701 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,701 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:02,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:02,702 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:02,702 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:02,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:02,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,702 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:02,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:02,703 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@559dac02[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:02,703 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:02,703 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,704 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,704 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,704 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:02,704 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:02,704 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:02,704 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,704 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:02,705 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,705 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:02,705 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:02,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:02,705 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:02,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:02,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:02,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:02,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:02,708 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:02,708 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:02,708 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:02,708 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:02,708 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:02,708 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:02,709 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,709 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,709 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:02,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:02,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:02,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:02,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:02,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:02,710 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:02,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,711 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:02,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,714 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,714 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:02,714 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:02,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:02,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:02,714 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:02,714 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:02,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:02,714 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:02,714 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:02,715 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,715 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,715 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,715 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,715 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:02,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:02,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:02,715 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:02,715 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:02,752 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:02,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 201 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53126 deadline: 1680922562750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,815 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:02,816 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:02,817 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:02,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:02,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:02,818 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:02,818 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:02,818 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:02,818 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:02,819 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,819 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:02,822 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,822 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:02,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:02,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:02,823 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,823 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:02,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:02,823 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:02,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:02,824 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:02,824 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:02,824 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:02,824 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:02,824 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:02,824 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:02,824 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:02,824 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:02,824 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:02,824 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:55:02,824 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:55:02,824 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:02,825 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:02,825 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:02,825 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,825 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:02,825 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:02,826 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,826 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:02,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:02,826 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,826 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:02,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:02,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:02,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:02,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:02,828 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@522ef8a9[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:02,828 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:02,828 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,829 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,829 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:02,829 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:02,829 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,829 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:02,829 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:02,831 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,831 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:02,831 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:02,832 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:02,832 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:02,832 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:02,833 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:02,834 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:02,834 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:02,834 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:02,834 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:02,834 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:02,834 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:02,836 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,836 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:02,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:02,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,836 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:02,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:02,836 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,836 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:02,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:02,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:02,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,837 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:02,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,839 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,839 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:02,840 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:02,840 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:02,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:02,840 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:02,840 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,840 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,840 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,840 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:02,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:02,840 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:02,840 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:02,840 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:02,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:02,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:02,841 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:02,841 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:02,854 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:02,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 203 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53126 deadline: 1680922562853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,941 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:02,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:02,943 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:02,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:02,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:02,945 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:02,945 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:02,945 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:02,945 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:02,945 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,945 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:02,946 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:02,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:02,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:02,947 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:02,947 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,947 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:02,947 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:02,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:02,948 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:02,948 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:02,948 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:02,948 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:02,948 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:02,948 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:02,948 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:02,948 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:02,948 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:02,949 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:55:02,949 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:55:02,949 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:02,949 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:02,949 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:02,949 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,949 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:02,949 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:02,952 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,952 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:02,952 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,952 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:02,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:02,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:02,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:02,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d530502[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:02,954 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:02,954 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,956 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,956 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:02,956 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:02,956 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:02,956 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,956 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:02,957 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,957 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:02,957 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:02,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:02,957 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:02,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:02,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:02,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:02,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:02,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:02,960 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:02,960 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:02,960 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:02,960 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:02,960 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:02,961 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,961 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,961 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:02,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:02,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:02,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:02,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:02,962 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:02,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:02,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,963 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:02,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:02,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,966 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:02,966 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:02,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:02,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:02,966 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,967 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:02,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:02,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:02,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:02,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:02,967 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:02,967 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:02,967 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:02,967 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:02,967 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:02,967 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:02,967 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,967 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:02,967 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,059 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:03,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53126 deadline: 1680922563056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,067 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:03,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:03,069 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:03,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:03,070 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:03,071 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:03,071 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:03,071 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:03,071 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:03,071 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,071 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,072 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:03,072 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,072 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:03,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:03,073 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,073 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:03,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,077 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,077 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=531 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/d743cc053e1442e6aa2f4a75916ae62a 2023-04-08 02:55:03,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:03,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:03,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:03,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:03,078 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:03,078 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:03,078 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:03,078 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:03,078 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:03,078 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:03,078 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:55:03,078 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:55:03,078 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:03,078 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:03,079 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:03,079 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,079 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:03,079 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:03,080 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,080 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:03,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:03,080 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,080 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:03,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:03,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:03,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,081 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/ce9ac13585d84af1a17ac92f23862440 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/ce9ac13585d84af1a17ac92f23862440 2023-04-08 02:55:03,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:03,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:03,082 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5a638a57[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:03,082 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:03,082 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,083 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,084 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:03,084 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:03,084 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:03,084 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,084 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:03,085 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/ce9ac13585d84af1a17ac92f23862440, entries=150, sequenceid=531, filesize=11.9 K 2023-04-08 02:55:03,086 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/3a39b47b08774b60874ff633656440f0 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/3a39b47b08774b60874ff633656440f0 2023-04-08 02:55:03,086 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:03,087 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,087 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:03,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:03,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:03,087 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:03,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:03,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:03,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:03,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:03,089 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:03,089 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:03,089 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:03,089 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:03,089 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:03,089 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:03,089 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/3a39b47b08774b60874ff633656440f0, entries=150, sequenceid=531, filesize=11.9 K 2023-04-08 02:55:03,090 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/d743cc053e1442e6aa2f4a75916ae62a as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/d743cc053e1442e6aa2f4a75916ae62a 2023-04-08 02:55:03,091 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,091 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,091 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:03,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,091 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:03,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:03,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:03,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:03,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,093 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:03,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,094 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/d743cc053e1442e6aa2f4a75916ae62a, entries=150, sequenceid=531, filesize=11.9 K 2023-04-08 02:55:03,095 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,095 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:03,095 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,095 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:03,095 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:03,095 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:03,095 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for d3e62a2534068dd21a6b490550cac850 in 458ms, sequenceid=531, compaction requested=true 2023-04-08 02:55:03,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:03,095 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:03,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:03,095 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:03,095 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,095 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:03,095 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:03,095 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,095 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:03,096 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:03,096 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:55:03,096 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:03,096 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,096 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-08 02:55:03,096 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:03,096 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:03,096 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,096 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:55:03,097 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49902 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:55:03,097 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/A is initiating minor compaction (all files) 2023-04-08 02:55:03,097 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/A in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:03,097 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/ae982203a8da425b8d9333063dc7ef8f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/a4c20daf19284992b2f78e4224508266, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/5effe2bd68f54744a2c86cc12cbe380f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/ce9ac13585d84af1a17ac92f23862440] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=48.7 K 2023-04-08 02:55:03,098 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49902 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:55:03,098 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting ae982203a8da425b8d9333063dc7ef8f, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1680922499659 2023-04-08 02:55:03,098 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/B is initiating minor compaction (all files) 2023-04-08 02:55:03,098 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/B in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:03,098 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/65afa1e381f04a04ba887afd9f533488, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/ec5c11b9f14441d5ab5a9b71f185b4fc, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/39322bf082634a479465bdac26ec9df6, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/3a39b47b08774b60874ff633656440f0] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=48.7 K 2023-04-08 02:55:03,098 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting a4c20daf19284992b2f78e4224508266, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1680922500844 2023-04-08 02:55:03,099 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 65afa1e381f04a04ba887afd9f533488, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1680922499659 2023-04-08 02:55:03,099 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 5effe2bd68f54744a2c86cc12cbe380f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1680922501492 2023-04-08 02:55:03,099 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting ec5c11b9f14441d5ab5a9b71f185b4fc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1680922500844 2023-04-08 02:55:03,099 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting ce9ac13585d84af1a17ac92f23862440, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1680922502627 2023-04-08 02:55:03,099 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 39322bf082634a479465bdac26ec9df6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1680922501492 2023-04-08 02:55:03,101 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 3a39b47b08774b60874ff633656440f0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1680922502627 2023-04-08 02:55:03,108 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#A#compaction#323 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:03,110 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#B#compaction#324 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:03,116 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/e898f1996830449ab32171a289f80645 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/e898f1996830449ab32171a289f80645 2023-04-08 02:55:03,120 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d3e62a2534068dd21a6b490550cac850/A of d3e62a2534068dd21a6b490550cac850 into e898f1996830449ab32171a289f80645(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:03,120 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:03,120 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/A, priority=12, startTime=1680922503096; duration=0sec 2023-04-08 02:55:03,120 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:03,120 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:55:03,122 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49902 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:55:03,122 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/C is initiating minor compaction (all files) 2023-04-08 02:55:03,122 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/C in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:03,122 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/d12e7e6f606a4a70892bcc65bfdf1bbf, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/f7167b5985104042963f2f27701c8c71, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/651757c81ba94e7ab7554ea4a3688447, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/d743cc053e1442e6aa2f4a75916ae62a] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=48.7 K 2023-04-08 02:55:03,122 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting d12e7e6f606a4a70892bcc65bfdf1bbf, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=485, earliestPutTs=1680922499659 2023-04-08 02:55:03,123 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting f7167b5985104042963f2f27701c8c71, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=504, earliestPutTs=1680922500844 2023-04-08 02:55:03,123 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 651757c81ba94e7ab7554ea4a3688447, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=520, earliestPutTs=1680922501492 2023-04-08 02:55:03,123 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting d743cc053e1442e6aa2f4a75916ae62a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1680922502627 2023-04-08 02:55:03,130 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#C#compaction#325 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:03,196 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:03,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:03,198 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:03,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:03,199 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:03,199 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:03,199 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:03,199 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:03,199 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:03,199 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,199 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,201 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,201 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:03,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:03,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:03,201 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,201 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:03,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,201 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:03,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,202 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:03,202 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:03,202 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:03,202 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:03,202 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:03,202 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:03,202 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:03,202 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:03,202 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:03,203 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:55:03,203 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-08 02:55:03,203 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:55:03,203 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:03,203 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:55:03,203 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:03,203 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:55:03,203 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:03,214 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=569 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/86f80b4295b84a7686944a21f681ede8 2023-04-08 02:55:03,228 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=569 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/d1f5ee65227d4855953ecd8a4a854aeb 2023-04-08 02:55:03,238 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=569 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/3f5b07e818e442b2953b4375da32a024 2023-04-08 02:55:03,242 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/86f80b4295b84a7686944a21f681ede8 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/86f80b4295b84a7686944a21f681ede8 2023-04-08 02:55:03,245 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/86f80b4295b84a7686944a21f681ede8, entries=150, sequenceid=569, filesize=11.9 K 2023-04-08 02:55:03,246 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/d1f5ee65227d4855953ecd8a4a854aeb as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d1f5ee65227d4855953ecd8a4a854aeb 2023-04-08 02:55:03,249 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d1f5ee65227d4855953ecd8a4a854aeb, entries=150, sequenceid=569, filesize=11.9 K 2023-04-08 02:55:03,250 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/3f5b07e818e442b2953b4375da32a024 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/3f5b07e818e442b2953b4375da32a024 2023-04-08 02:55:03,253 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/3f5b07e818e442b2953b4375da32a024, entries=150, sequenceid=569, filesize=11.9 K 2023-04-08 02:55:03,254 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=0 B/0 for d3e62a2534068dd21a6b490550cac850 in 51ms, sequenceid=569, compaction requested=false 2023-04-08 02:55:03,254 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:03,254 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:03,254 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:03,254 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:03,254 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,254 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:03,254 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:03,258 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:03,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:03,258 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:03,258 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,258 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:03,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:03,260 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@62a6d0a5[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:03,260 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:03,260 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,262 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,263 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:03,263 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,263 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:03,263 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:03,263 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:03,264 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:03,264 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:03,264 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,264 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:03,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:03,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:03,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:03,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:03,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:03,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:03,267 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:03,267 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:03,267 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:03,267 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:03,267 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:03,268 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:03,269 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,269 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:03,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:03,269 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:03,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:03,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:03,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:03,270 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:03,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,271 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:03,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,274 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:03,274 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:03,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:03,274 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:03,274 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:03,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:03,274 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:03,274 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,275 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,275 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,275 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:03,274 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:03,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:03,275 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,275 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:03,275 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,275 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:55:03,374 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-08 02:55:03,374 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:55:03,374 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:03,374 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:55:03,374 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:03,374 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:55:03,374 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:03,375 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:03,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:03,378 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:03,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:03,379 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:03,379 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:03,379 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:03,379 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:03,379 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:03,379 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,380 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,382 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:03,382 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,382 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:03,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:03,382 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,382 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:03,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,383 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:03,383 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:03,383 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:03,383 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:03,383 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:03,383 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:03,384 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:03,384 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:03,384 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:03,384 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:55:03,384 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:55:03,384 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:03,384 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:03,384 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:03,384 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,384 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:03,384 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:03,386 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,386 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:03,386 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,386 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:03,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:03,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:03,387 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@47e897e8[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:03,387 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:03,387 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,388 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,389 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:03,389 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:03,389 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:03,389 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,389 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:03,390 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,390 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:03,390 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:03,390 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:03,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:03,392 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:03,393 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:03,393 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:03,393 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:03,393 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:03,395 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,395 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,395 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:03,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:03,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:03,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:03,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:03,396 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:03,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,397 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:03,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,399 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,399 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,399 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,399 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,399 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:03,399 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:03,399 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,399 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:03,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:03,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:03,399 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:03,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:03,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:03,399 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,399 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,399 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:03,400 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:03,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:03,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:03,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922503405, max timestamp=1680922503429Num uniques 859; 2023-04-08 02:55:03,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922503405, max timestamp=1680922503429Num uniques 859; 2023-04-08 02:55:03,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:55:03,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922503405, max timestamp=1680922503429Num uniques 859; 2023-04-08 02:55:03,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:55:03,431 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:55:03,490 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:03,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53126 deadline: 1680922563488, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,500 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:03,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:03,501 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:03,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:03,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:03,503 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:03,503 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:03,503 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:03,503 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:03,503 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,503 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,504 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:03,504 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,504 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:03,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:03,505 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,505 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,505 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,505 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:03,506 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:03,506 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:03,506 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:03,506 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:03,506 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:03,506 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:03,506 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:03,506 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:55:03,506 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:55:03,506 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:03,506 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:03,506 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:03,507 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,507 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:03,507 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:03,508 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:03,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:03,509 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:03,510 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:03,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5286f102[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:03,510 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,512 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,512 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:03,512 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:03,512 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:03,512 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,512 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:03,513 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,513 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:03,513 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:03,514 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:03,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:03,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:03,516 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:03,516 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:03,516 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:03,516 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:03,516 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:03,516 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:03,518 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,518 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,518 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:03,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:03,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:03,519 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/8159382b9f5b4c3196b652c3a781af89 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/8159382b9f5b4c3196b652c3a781af89 2023-04-08 02:55:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,519 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,521 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:03,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,523 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,523 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,523 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,523 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:03,524 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d3e62a2534068dd21a6b490550cac850/B of d3e62a2534068dd21a6b490550cac850 into 8159382b9f5b4c3196b652c3a781af89(size=13.2 K), total size for store is 25.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:03,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:03,524 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:03,523 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:03,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:03,523 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,524 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,524 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,524 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:03,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:03,524 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:03,524 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:03,524 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:03,524 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/B, priority=12, startTime=1680922503096; duration=0sec 2023-04-08 02:55:03,524 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:03,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:03,524 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:03,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:03,541 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/7d800ad830ee4e8889ad407b1b6f6ad1 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/7d800ad830ee4e8889ad407b1b6f6ad1 2023-04-08 02:55:03,546 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in d3e62a2534068dd21a6b490550cac850/C of d3e62a2534068dd21a6b490550cac850 into 7d800ad830ee4e8889ad407b1b6f6ad1(size=13.2 K), total size for store is 25.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:03,546 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:03,546 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/C, priority=12, startTime=1680922503096; duration=0sec 2023-04-08 02:55:03,546 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:03,592 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:03,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53126 deadline: 1680922563591, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,624 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:03,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:03,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:03,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:03,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:03,627 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:03,628 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:03,628 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:03,628 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:03,628 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,628 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,631 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:03,631 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:03,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:03,631 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,631 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:03,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,631 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:03,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,632 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:03,632 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:03,632 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:03,632 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:03,632 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:03,632 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:03,632 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:03,633 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:03,633 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:03,633 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:55:03,633 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:55:03,633 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:03,633 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:03,633 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:03,633 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,633 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:03,633 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:03,634 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,634 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:03,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:03,635 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,635 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:03,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:03,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:03,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:03,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:03,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1471f837[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:03,636 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:03,636 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,637 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,637 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,637 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:03,638 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:03,637 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:03,638 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:03,643 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,643 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:03,643 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:03,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:03,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:03,643 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:03,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:03,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:03,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:03,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:03,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:03,645 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:03,645 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:03,645 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:03,645 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:03,645 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:03,647 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,647 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:03,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:03,647 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:03,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:03,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:03,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:03,648 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:03,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,649 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:03,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,651 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,652 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:03,652 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:03,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:03,652 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,652 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:03,652 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:03,652 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:03,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:03,652 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:03,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:03,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:03,652 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,652 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,652 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:03,652 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:03,753 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:03,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:03,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:03,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:03,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:03,756 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:03,756 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:03,756 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:03,756 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:03,756 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,756 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,757 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,757 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:03,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:03,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:03,758 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,758 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:03,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:03,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:03,759 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:03,759 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:03,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:03,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:03,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:03,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:03,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:03,759 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:03,759 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:55:03,759 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:55:03,759 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:03,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:03,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:03,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,760 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:03,760 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:03,762 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,762 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:03,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:03,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:03,763 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,763 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:03,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:03,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:03,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:03,765 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:03,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@408631e4[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:03,765 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,766 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,766 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,766 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:03,766 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:03,766 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:03,766 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,766 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:03,768 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:03,768 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:03,768 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,768 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:03,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:03,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:03,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:03,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:03,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:03,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:03,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:03,771 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:03,771 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:03,771 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:03,771 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:03,771 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:03,773 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,773 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:03,773 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,774 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:03,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:03,774 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:03,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:03,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:03,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:03,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:03,778 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:03,778 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:03,778 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,778 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:03,778 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:03,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:03,778 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:03,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:03,778 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:03,779 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:03,779 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:03,779 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,779 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,779 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:03,779 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:03,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:03,780 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:03,783 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/6742d944eaae49868eb7fd69960dce9f 2023-04-08 02:55:03,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:03,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53126 deadline: 1680922563797, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,819 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/ade5ddc28d094489884c062305cb7eff 2023-04-08 02:55:03,846 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=579 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/0b3d61a800c84e60914595f87ff27192 2023-04-08 02:55:03,850 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/6742d944eaae49868eb7fd69960dce9f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/6742d944eaae49868eb7fd69960dce9f 2023-04-08 02:55:03,855 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/6742d944eaae49868eb7fd69960dce9f, entries=150, sequenceid=579, filesize=11.9 K 2023-04-08 02:55:03,856 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/ade5ddc28d094489884c062305cb7eff as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/ade5ddc28d094489884c062305cb7eff 2023-04-08 02:55:03,861 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/ade5ddc28d094489884c062305cb7eff, entries=150, sequenceid=579, filesize=11.9 K 2023-04-08 02:55:03,862 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/0b3d61a800c84e60914595f87ff27192 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/0b3d61a800c84e60914595f87ff27192 2023-04-08 02:55:03,865 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/0b3d61a800c84e60914595f87ff27192, entries=150, sequenceid=579, filesize=11.9 K 2023-04-08 02:55:03,865 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for d3e62a2534068dd21a6b490550cac850 in 491ms, sequenceid=579, compaction requested=true 2023-04-08 02:55:03,866 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:03,866 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:03,866 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:55:03,866 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:55:03,866 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:03,866 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:03,867 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:55:03,867 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:55:03,867 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/A is initiating minor compaction (all files) 2023-04-08 02:55:03,867 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/B is initiating minor compaction (all files) 2023-04-08 02:55:03,867 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/A in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:03,867 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/B in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:03,867 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/e898f1996830449ab32171a289f80645, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/86f80b4295b84a7686944a21f681ede8, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/6742d944eaae49868eb7fd69960dce9f] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=37.0 K 2023-04-08 02:55:03,867 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/8159382b9f5b4c3196b652c3a781af89, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d1f5ee65227d4855953ecd8a4a854aeb, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/ade5ddc28d094489884c062305cb7eff] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=37.0 K 2023-04-08 02:55:03,867 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting e898f1996830449ab32171a289f80645, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1680922502627 2023-04-08 02:55:03,867 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 8159382b9f5b4c3196b652c3a781af89, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1680922502627 2023-04-08 02:55:03,868 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 86f80b4295b84a7686944a21f681ede8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=569, earliestPutTs=1680922502735 2023-04-08 02:55:03,868 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting d1f5ee65227d4855953ecd8a4a854aeb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=569, earliestPutTs=1680922502735 2023-04-08 02:55:03,868 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 6742d944eaae49868eb7fd69960dce9f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1680922503365 2023-04-08 02:55:03,868 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting ade5ddc28d094489884c062305cb7eff, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1680922503365 2023-04-08 02:55:03,876 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#B#compaction#332 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:03,876 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#A#compaction#333 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:03,879 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:03,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:03,881 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:03,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:03,884 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:03,884 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:03,884 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:03,884 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:03,884 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:03,885 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,885 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,886 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:03,886 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:03,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:03,887 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:03,887 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:03,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,887 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:03,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:03,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:03,888 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:03,888 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:03,888 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:03,888 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/25e497722d1c42e5b70ea3511dd7f5f2 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/25e497722d1c42e5b70ea3511dd7f5f2 2023-04-08 02:55:03,888 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:03,889 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:03,889 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:03,889 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:03,889 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:03,889 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:03,889 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:55:03,889 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-08 02:55:03,889 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:55:03,889 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:03,889 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:55:03,889 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:03,890 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:55:03,890 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:03,894 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/B of d3e62a2534068dd21a6b490550cac850 into 25e497722d1c42e5b70ea3511dd7f5f2(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:03,894 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:03,894 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/B, priority=13, startTime=1680922503866; duration=0sec 2023-04-08 02:55:03,894 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:03,894 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:55:03,895 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37852 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:55:03,895 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/C is initiating minor compaction (all files) 2023-04-08 02:55:03,895 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/C in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:03,895 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/7d800ad830ee4e8889ad407b1b6f6ad1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/3f5b07e818e442b2953b4375da32a024, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/0b3d61a800c84e60914595f87ff27192] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=37.0 K 2023-04-08 02:55:03,896 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 7d800ad830ee4e8889ad407b1b6f6ad1, keycount=150, bloomtype=ROW, size=13.2 K, encoding=NONE, compression=NONE, seqNum=531, earliestPutTs=1680922502627 2023-04-08 02:55:03,896 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 3f5b07e818e442b2953b4375da32a024, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=569, earliestPutTs=1680922502735 2023-04-08 02:55:03,896 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 0b3d61a800c84e60914595f87ff27192, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1680922503365 2023-04-08 02:55:03,900 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=618 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/775e58a24687413c8c7c0a8f6b7b091e 2023-04-08 02:55:03,904 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#C#compaction#335 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:04,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:55:04,104 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:55:04,120 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:04,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 261 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53126 deadline: 1680922564119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,222 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:04,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 263 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53126 deadline: 1680922564221, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,290 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/2f7011414c594a1f87355524b193a5a1 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/2f7011414c594a1f87355524b193a5a1 2023-04-08 02:55:04,294 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/A of d3e62a2534068dd21a6b490550cac850 into 2f7011414c594a1f87355524b193a5a1(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:04,294 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:04,294 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/A, priority=13, startTime=1680922503866; duration=0sec 2023-04-08 02:55:04,294 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:04,320 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/77cad5ab77b3478db8c6a18972166e02 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/77cad5ab77b3478db8c6a18972166e02 2023-04-08 02:55:04,324 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/C of d3e62a2534068dd21a6b490550cac850 into 77cad5ab77b3478db8c6a18972166e02(size=13.3 K), total size for store is 13.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:04,324 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:04,324 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/C, priority=13, startTime=1680922503866; duration=0sec 2023-04-08 02:55:04,324 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:04,328 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=618 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/afe7fafe57d94ed888002440028059e8 2023-04-08 02:55:04,339 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=618 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/a258fb818fb84050805adb9555bbe966 2023-04-08 02:55:04,343 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/775e58a24687413c8c7c0a8f6b7b091e as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/775e58a24687413c8c7c0a8f6b7b091e 2023-04-08 02:55:04,347 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/775e58a24687413c8c7c0a8f6b7b091e, entries=150, sequenceid=618, filesize=11.9 K 2023-04-08 02:55:04,348 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/afe7fafe57d94ed888002440028059e8 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/afe7fafe57d94ed888002440028059e8 2023-04-08 02:55:04,351 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/afe7fafe57d94ed888002440028059e8, entries=150, sequenceid=618, filesize=11.9 K 2023-04-08 02:55:04,352 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/a258fb818fb84050805adb9555bbe966 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/a258fb818fb84050805adb9555bbe966 2023-04-08 02:55:04,356 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/a258fb818fb84050805adb9555bbe966, entries=150, sequenceid=618, filesize=11.9 K 2023-04-08 02:55:04,357 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for d3e62a2534068dd21a6b490550cac850 in 468ms, sequenceid=618, compaction requested=false 2023-04-08 02:55:04,357 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:04,357 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:04,357 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:04,357 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:04,357 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,357 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:04,357 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:04,358 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,358 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:04,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:04,359 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,359 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:04,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:04,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:04,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:04,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:04,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:04,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@43d2b9cb[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:04,360 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:04,361 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,362 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,362 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:04,362 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:04,362 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:04,362 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,362 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:04,364 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,364 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:04,364 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:04,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:04,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:04,365 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:04,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:04,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:04,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:04,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:04,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:04,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:04,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:04,367 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:04,367 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:04,367 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:04,367 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:04,367 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:04,369 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,369 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,369 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:04,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:04,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:04,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,369 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:04,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:04,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:04,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:04,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:04,371 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:04,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:04,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:04,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,373 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,373 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:04,373 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:04,373 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,373 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:04,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:04,373 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:04,373 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:04,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:04,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:04,373 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,373 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,373 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:04,374 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:04,374 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:04,373 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:04,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:04,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:04,426 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:55:04,427 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-08 02:55:04,427 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:55:04,427 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:04,427 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:55:04,427 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:04,427 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:55:04,427 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:04,436 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=632 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/9081ac96701242ac922110ea76bafa22 2023-04-08 02:55:04,456 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=632 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/341bad5545e143e298e5baeccde2d88d 2023-04-08 02:55:04,474 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:04,474 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:04,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:04,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:04,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:04,477 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:04,477 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:04,477 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:04,477 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:04,477 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,477 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:04,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922504462, max timestamp=1680922504476Num uniques 859; 2023-04-08 02:55:04,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922504462, max timestamp=1680922504476Num uniques 859; 2023-04-08 02:55:04,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:55:04,478 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:55:04,478 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,479 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,479 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:04,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922504462, max timestamp=1680922504476Num uniques 859; 2023-04-08 02:55:04,479 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:55:04,479 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:04,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:04,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:04,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:04,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:04,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:04,480 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:04,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:04,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:04,480 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:04,480 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:04,480 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:04,481 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:04,481 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:04,481 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:04,481 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:55:04,481 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:55:04,481 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:04,481 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:04,481 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:04,481 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,481 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:04,481 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:04,484 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,484 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:04,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:04,484 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,484 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:04,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:04,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:04,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:04,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:04,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:04,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26b95e52[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:04,486 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:04,486 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,487 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,487 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,487 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:04,487 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:04,487 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:04,488 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:04,488 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:04,488 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,488 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:04,489 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:04,489 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:04,489 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:04,489 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:04,489 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:04,489 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:04,489 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:04,494 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,494 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:04,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:04,494 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:04,494 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:04,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:04,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:04,495 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:04,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:04,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:04,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,498 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,498 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:04,498 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:04,498 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:04,498 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:04,498 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,498 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,498 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,498 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,498 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:04,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:04,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:04,498 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:04,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:04,498 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:04,499 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:04,499 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:04,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:04,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:04,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:04,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 299 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53126 deadline: 1680922564521, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,599 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:04,599 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:04,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:04,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:04,602 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:04,602 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:04,602 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:04,602 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:04,602 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:04,603 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,603 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:04,604 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:04,604 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,604 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:04,604 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:04,605 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:04,605 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:04,605 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:04,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:04,605 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:04,606 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:04,606 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:04,606 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:04,606 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:04,606 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:04,606 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:04,606 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:04,606 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:04,606 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:55:04,606 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:55:04,606 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:04,606 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:04,606 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:04,606 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,606 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:04,606 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:04,608 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,608 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:04,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:04,608 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,608 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:04,608 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:04,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:04,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:04,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:04,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:04,610 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d12aaf2[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:04,610 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:04,610 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,612 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,612 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,612 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,612 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:04,612 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:04,612 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,612 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:04,612 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:04,613 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:04,614 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:04,613 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,614 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:04,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:04,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:04,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:04,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:04,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:04,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:04,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:04,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:04,616 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:04,616 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:04,616 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:04,616 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:04,616 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:04,616 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:04,618 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,618 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:04,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:04,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,618 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:04,618 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:04,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:04,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:04,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:04,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:04,619 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:04,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:04,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:04,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,622 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,622 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:04,622 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:04,622 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,622 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,622 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:04,622 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:04,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:04,622 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:04,622 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:04,622 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,622 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:04,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:04,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:04,622 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:04,622 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:04,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:04,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:04,624 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:04,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 301 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53126 deadline: 1680922564624, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,723 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:04,723 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:04,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:04,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:04,726 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:04,726 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:04,726 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:04,726 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:04,726 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:04,727 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,727 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:04,728 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:04,728 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:04,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:04,729 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,729 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:04,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:04,730 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:04,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:04,730 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:04,731 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:04,731 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:04,731 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:04,731 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:04,731 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:04,731 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:04,731 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:04,731 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:04,731 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:55:04,731 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:55:04,731 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:04,731 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:04,731 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:04,732 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,732 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:04,732 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:04,733 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:04,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:04,734 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:04,734 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,734 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:04,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:04,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:04,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:04,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:04,735 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4c6ab5a5[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:04,735 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:04,736 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,737 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,737 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,737 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,737 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:04,737 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:04,737 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:04,737 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,737 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:04,738 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:04,738 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,738 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:04,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:04,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:04,739 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:04,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:04,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:04,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:04,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:04,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:04,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:04,742 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:04,742 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:04,742 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:04,742 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:04,742 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:04,742 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:04,745 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,745 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:04,745 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,746 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:04,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:04,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,747 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,748 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,750 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:04,750 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:04,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:04,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:04,751 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:04,751 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,751 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:04,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:04,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:04,751 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:04,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:04,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:04,751 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:04,751 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:04,752 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,752 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,752 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,752 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:04,751 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:04,752 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:04,752 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:04,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:04,753 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:04,827 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:04,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53126 deadline: 1680922564825, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,852 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:04,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:04,855 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:04,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:04,876 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=632 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/62185f6e01e94ab49f528f76fca40362 2023-04-08 02:55:04,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:04,876 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:04,876 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:04,876 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:04,876 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:04,877 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,877 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:04,878 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:04,878 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:04,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:04,878 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,878 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:04,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:04,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:04,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:04,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:04,884 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/9081ac96701242ac922110ea76bafa22 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/9081ac96701242ac922110ea76bafa22 2023-04-08 02:55:04,888 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/9081ac96701242ac922110ea76bafa22, entries=150, sequenceid=632, filesize=11.9 K 2023-04-08 02:55:04,891 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/341bad5545e143e298e5baeccde2d88d as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/341bad5545e143e298e5baeccde2d88d 2023-04-08 02:55:04,892 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:04,892 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:04,892 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:04,892 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:04,892 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:04,893 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:04,893 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:04,893 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:04,893 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:55:04,893 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:55:04,893 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:04,893 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:04,893 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:04,893 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,893 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:04,893 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:04,895 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:04,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:04,895 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,895 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:04,895 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:04,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:04,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:04,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:04,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:04,898 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1aa04102[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:04,898 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:04,898 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,898 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/341bad5545e143e298e5baeccde2d88d, entries=150, sequenceid=632, filesize=11.9 K 2023-04-08 02:55:04,899 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/62185f6e01e94ab49f528f76fca40362 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/62185f6e01e94ab49f528f76fca40362 2023-04-08 02:55:04,901 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,901 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:04,901 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:04,901 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:04,902 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,902 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:04,903 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:04,903 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:04,903 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:04,903 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:04,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:04,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:04,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:04,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:04,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,905 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/62185f6e01e94ab49f528f76fca40362, entries=150, sequenceid=632, filesize=11.9 K 2023-04-08 02:55:04,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:04,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:04,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,906 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for d3e62a2534068dd21a6b490550cac850 in 480ms, sequenceid=632, compaction requested=true 2023-04-08 02:55:04,906 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:04,906 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:04,906 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:55:04,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:04,906 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:04,906 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:04,906 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:04,906 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:04,906 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:04,906 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:04,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:04,906 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:55:04,906 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:04,907 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37954 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:55:04,907 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/A is initiating minor compaction (all files) 2023-04-08 02:55:04,907 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/A in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:04,907 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/2f7011414c594a1f87355524b193a5a1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/775e58a24687413c8c7c0a8f6b7b091e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/9081ac96701242ac922110ea76bafa22] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=37.1 K 2023-04-08 02:55:04,908 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 2f7011414c594a1f87355524b193a5a1, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1680922503365 2023-04-08 02:55:04,908 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37954 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:55:04,908 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/B is initiating minor compaction (all files) 2023-04-08 02:55:04,908 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/B in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:04,908 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 775e58a24687413c8c7c0a8f6b7b091e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=618, earliestPutTs=1680922503480 2023-04-08 02:55:04,908 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/25e497722d1c42e5b70ea3511dd7f5f2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/afe7fafe57d94ed888002440028059e8, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/341bad5545e143e298e5baeccde2d88d] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=37.1 K 2023-04-08 02:55:04,909 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,909 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:04,909 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,909 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 9081ac96701242ac922110ea76bafa22, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=632, earliestPutTs=1680922504112 2023-04-08 02:55:04,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:04,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:04,909 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 25e497722d1c42e5b70ea3511dd7f5f2, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1680922503365 2023-04-08 02:55:04,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:04,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:04,910 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting afe7fafe57d94ed888002440028059e8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=618, earliestPutTs=1680922503480 2023-04-08 02:55:04,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:04,910 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 341bad5545e143e298e5baeccde2d88d, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=632, earliestPutTs=1680922504112 2023-04-08 02:55:04,910 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:04,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:04,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:04,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,911 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:04,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:04,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,924 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:04,925 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,925 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:04,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:04,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:04,925 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:04,925 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:04,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:04,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:04,925 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:04,925 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:04,925 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,926 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:04,926 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:04,926 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:04,926 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:04,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:04,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:04,926 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#A#compaction#341 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:04,937 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#B#compaction#342 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:04,964 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/0870d9d29e69485d8fe598d942c1a3d2 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/0870d9d29e69485d8fe598d942c1a3d2 2023-04-08 02:55:04,969 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/A of d3e62a2534068dd21a6b490550cac850 into 0870d9d29e69485d8fe598d942c1a3d2(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:04,969 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:04,969 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/A, priority=13, startTime=1680922504906; duration=0sec 2023-04-08 02:55:04,969 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:04,969 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:55:04,971 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37954 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:55:04,971 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/C is initiating minor compaction (all files) 2023-04-08 02:55:04,971 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/C in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:04,971 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/77cad5ab77b3478db8c6a18972166e02, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/a258fb818fb84050805adb9555bbe966, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/62185f6e01e94ab49f528f76fca40362] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=37.1 K 2023-04-08 02:55:04,971 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 77cad5ab77b3478db8c6a18972166e02, keycount=150, bloomtype=ROW, size=13.3 K, encoding=NONE, compression=NONE, seqNum=579, earliestPutTs=1680922503365 2023-04-08 02:55:04,972 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting a258fb818fb84050805adb9555bbe966, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=618, earliestPutTs=1680922503480 2023-04-08 02:55:04,972 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 62185f6e01e94ab49f528f76fca40362, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=632, earliestPutTs=1680922504112 2023-04-08 02:55:04,974 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/27774f5acf3a42c09467f19e3498b034 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/27774f5acf3a42c09467f19e3498b034 2023-04-08 02:55:04,978 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#C#compaction#343 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:04,981 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/B of d3e62a2534068dd21a6b490550cac850 into 27774f5acf3a42c09467f19e3498b034(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:04,981 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:04,981 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/B, priority=13, startTime=1680922504906; duration=0sec 2023-04-08 02:55:04,981 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:05,026 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:05,026 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:05,028 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:05,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:05,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:05,029 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:05,029 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:05,029 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:05,029 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:05,030 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:05,030 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:05,031 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:05,031 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:05,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:05,032 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,032 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:05,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:05,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:05,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:05,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:05,033 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:05,033 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:05,033 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:05,033 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:05,033 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:05,033 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:05,033 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:05,034 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:05,034 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:05,034 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:55:05,034 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-08 02:55:05,034 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:55:05,034 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:05,034 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:55:05,034 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:05,034 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:55:05,034 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:05,058 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=670 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/692f290bf50c402494cced7d7e14d30a 2023-04-08 02:55:05,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:55:05,130 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:55:05,154 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:05,154 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 313 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53126 deadline: 1680922565153, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,256 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:05,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 315 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53126 deadline: 1680922565255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,391 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/64e83a5d70ba49f38492b3bdeefd6239 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/64e83a5d70ba49f38492b3bdeefd6239 2023-04-08 02:55:05,403 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/C of d3e62a2534068dd21a6b490550cac850 into 64e83a5d70ba49f38492b3bdeefd6239(size=13.4 K), total size for store is 13.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:05,403 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:05,403 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/C, priority=13, startTime=1680922504906; duration=0sec 2023-04-08 02:55:05,403 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:05,460 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:05,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:53126 deadline: 1680922565457, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,471 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=670 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/37527ab0dfdf459aa682fcc1089ba805 2023-04-08 02:55:05,518 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=670 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/09395a24ac11431997f2ccdd58fd6e97 2023-04-08 02:55:05,523 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/692f290bf50c402494cced7d7e14d30a as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/692f290bf50c402494cced7d7e14d30a 2023-04-08 02:55:05,527 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/692f290bf50c402494cced7d7e14d30a, entries=150, sequenceid=670, filesize=11.9 K 2023-04-08 02:55:05,528 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/37527ab0dfdf459aa682fcc1089ba805 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/37527ab0dfdf459aa682fcc1089ba805 2023-04-08 02:55:05,542 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/37527ab0dfdf459aa682fcc1089ba805, entries=150, sequenceid=670, filesize=11.9 K 2023-04-08 02:55:05,543 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/09395a24ac11431997f2ccdd58fd6e97 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/09395a24ac11431997f2ccdd58fd6e97 2023-04-08 02:55:05,566 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/09395a24ac11431997f2ccdd58fd6e97, entries=150, sequenceid=670, filesize=11.9 K 2023-04-08 02:55:05,567 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for d3e62a2534068dd21a6b490550cac850 in 533ms, sequenceid=670, compaction requested=false 2023-04-08 02:55:05,567 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:05,567 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:05,567 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:05,567 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:05,567 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,567 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:05,567 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:05,569 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:05,569 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:05,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:05,570 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:05,570 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:05,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:05,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:05,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:05,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:05,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:05,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7607d7ca[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:05,571 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:05,571 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:05,572 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:05,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:05,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:05,573 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:05,573 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:05,573 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:05,573 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,573 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:05,576 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:05,576 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,576 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:05,576 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:05,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:05,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:05,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:05,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:05,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:05,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:05,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:05,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:05,579 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:05,579 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:05,579 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:05,580 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:05,580 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:05,580 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:05,581 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:05,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:05,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:05,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:05,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:05,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:05,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:05,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:05,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:05,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:05,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:05,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:05,583 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:05,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:05,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:05,584 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:05,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:05,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,587 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:05,587 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:05,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:05,587 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:05,587 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:05,587 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,587 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:05,587 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:05,587 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:05,587 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:05,587 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:05,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:05,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:05,587 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:05,588 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:05,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:05,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:05,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:05,688 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:05,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:05,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:05,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:05,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:05,694 DEBUG [Thread-1629] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x08cd7cb0 to 127.0.0.1:64826 2023-04-08 02:55:05,694 DEBUG [Thread-1637] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3b4658f8 to 127.0.0.1:64826 2023-04-08 02:55:05,694 DEBUG [Thread-1637] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:55:05,694 DEBUG [Thread-1629] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:55:05,696 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:05,696 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:05,696 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:05,696 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:05,696 DEBUG [Thread-1635] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66ee0bd0 to 127.0.0.1:64826 2023-04-08 02:55:05,697 DEBUG [Thread-1635] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:55:05,697 DEBUG [Thread-1633] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x433f251d to 127.0.0.1:64826 2023-04-08 02:55:05,697 DEBUG [Thread-1633] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:55:05,698 DEBUG [Thread-1631] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0ea1ca03 to 127.0.0.1:64826 2023-04-08 02:55:05,698 DEBUG [Thread-1631] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:55:05,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on d3e62a2534068dd21a6b490550cac850 2023-04-08 02:55:05,745 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-08 02:55:05,745 DEBUG [Thread-1622] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3157d4df to 127.0.0.1:64826 2023-04-08 02:55:05,745 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:55:05,745 DEBUG [Thread-1622] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:55:05,745 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:05,745 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:55:05,745 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:05,745 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:55:05,745 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:05,762 DEBUG [Thread-1618] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x23c97092 to 127.0.0.1:64826 2023-04-08 02:55:05,762 DEBUG [Thread-1618] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:55:05,900 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:05,900 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:05,901 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,901 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:05,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:05,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:05,902 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,902 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:05,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:05,902 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:05,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:05,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:05,902 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:05,903 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:05,903 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:05,903 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:05,903 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:05,903 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:05,903 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:05,903 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:05,903 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:05,903 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. started... 2023-04-08 02:55:05,903 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. as already flushing 2023-04-08 02:55:05,903 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:05,903 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:05,903 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:05,903 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,903 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:05,903 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:05,906 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,906 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:05,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:05,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:05,906 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:05,906 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:05,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:05,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:05,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:05,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:05,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:05,907 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5eeaa6c2[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:05,907 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:05,907 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:05,908 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:05,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:05,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:05,908 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:05,908 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:05,908 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,908 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:05,908 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:05,910 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,910 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:05,910 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:05,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:05,910 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:05,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:05,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:05,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:05,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:05,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:05,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:05,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:05,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:05,912 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:05,912 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:05,912 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:05,912 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:05,912 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:05,913 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:05,913 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:05,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:05,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:05,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:05,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:05,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:05,928 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:05,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:05,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:05,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:05,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:05,929 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:05,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:05,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:05,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,930 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:05,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:05,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,933 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,933 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:05,933 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:05,933 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:05,933 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:05,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:05,934 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:05,933 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:05,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:05,933 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:05,934 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:05,934 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:05,934 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:05,934 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:05,934 DEBUG [Thread-1628] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:05,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:05,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:05,934 DEBUG [Thread-1628] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:06,034 DEBUG [Thread-1628] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:06,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:06,152 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=683 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/161b783e52ac45a5b6b88f930e75810c 2023-04-08 02:55:06,560 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=683 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/bf27f2efdcd7449795a88556d5be8b88 2023-04-08 02:55:06,971 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=683 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/d688efb702e9428980791db7888938c7 2023-04-08 02:55:06,975 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/161b783e52ac45a5b6b88f930e75810c as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/161b783e52ac45a5b6b88f930e75810c 2023-04-08 02:55:06,978 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/161b783e52ac45a5b6b88f930e75810c, entries=100, sequenceid=683, filesize=9.5 K 2023-04-08 02:55:06,978 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/bf27f2efdcd7449795a88556d5be8b88 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/bf27f2efdcd7449795a88556d5be8b88 2023-04-08 02:55:06,981 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/bf27f2efdcd7449795a88556d5be8b88, entries=100, sequenceid=683, filesize=9.5 K 2023-04-08 02:55:06,982 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/d688efb702e9428980791db7888938c7 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/d688efb702e9428980791db7888938c7 2023-04-08 02:55:06,985 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/d688efb702e9428980791db7888938c7, entries=100, sequenceid=683, filesize=9.5 K 2023-04-08 02:55:06,986 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=6.71 KB/6870 for d3e62a2534068dd21a6b490550cac850 in 1241ms, sequenceid=683, compaction requested=true 2023-04-08 02:55:06,986 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:06,986 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:06,986 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:06,986 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:55:06,986 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:55:06,986 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:06,987 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35612 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:55:06,987 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35612 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:55:06,987 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/A is initiating minor compaction (all files) 2023-04-08 02:55:06,987 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/B is initiating minor compaction (all files) 2023-04-08 02:55:06,987 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/A in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:06,987 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/B in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:06,987 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/27774f5acf3a42c09467f19e3498b034, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/37527ab0dfdf459aa682fcc1089ba805, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/bf27f2efdcd7449795a88556d5be8b88] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=34.8 K 2023-04-08 02:55:06,987 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/0870d9d29e69485d8fe598d942c1a3d2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/692f290bf50c402494cced7d7e14d30a, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/161b783e52ac45a5b6b88f930e75810c] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=34.8 K 2023-04-08 02:55:06,988 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 27774f5acf3a42c09467f19e3498b034, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=632, earliestPutTs=1680922504112 2023-04-08 02:55:06,988 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 0870d9d29e69485d8fe598d942c1a3d2, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=632, earliestPutTs=1680922504112 2023-04-08 02:55:06,988 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 37527ab0dfdf459aa682fcc1089ba805, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1680922504512 2023-04-08 02:55:06,988 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 692f290bf50c402494cced7d7e14d30a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1680922504512 2023-04-08 02:55:06,988 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting bf27f2efdcd7449795a88556d5be8b88, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1680922505149 2023-04-08 02:55:06,988 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 161b783e52ac45a5b6b88f930e75810c, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1680922505149 2023-04-08 02:55:06,993 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#B#compaction#350 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:06,994 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#A#compaction#351 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:07,407 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/fbd98bf1d3a04733bf8f07f01aec24d1 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/fbd98bf1d3a04733bf8f07f01aec24d1 2023-04-08 02:55:07,407 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/8b1e229ee4c34594ac27eae2dad4bcea as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/8b1e229ee4c34594ac27eae2dad4bcea 2023-04-08 02:55:07,410 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/A of d3e62a2534068dd21a6b490550cac850 into fbd98bf1d3a04733bf8f07f01aec24d1(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:07,410 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/B of d3e62a2534068dd21a6b490550cac850 into 8b1e229ee4c34594ac27eae2dad4bcea(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:07,410 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:07,411 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:07,411 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/A, priority=13, startTime=1680922506986; duration=0sec 2023-04-08 02:55:07,411 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/B, priority=13, startTime=1680922506986; duration=0sec 2023-04-08 02:55:07,411 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:07,411 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:55:07,411 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:07,411 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35612 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:55:07,412 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): d3e62a2534068dd21a6b490550cac850/C is initiating minor compaction (all files) 2023-04-08 02:55:07,412 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of d3e62a2534068dd21a6b490550cac850/C in TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:07,412 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/64e83a5d70ba49f38492b3bdeefd6239, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/09395a24ac11431997f2ccdd58fd6e97, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/d688efb702e9428980791db7888938c7] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp, totalSize=34.8 K 2023-04-08 02:55:07,412 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 64e83a5d70ba49f38492b3bdeefd6239, keycount=150, bloomtype=ROW, size=13.4 K, encoding=NONE, compression=NONE, seqNum=632, earliestPutTs=1680922504112 2023-04-08 02:55:07,412 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 09395a24ac11431997f2ccdd58fd6e97, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=670, earliestPutTs=1680922504512 2023-04-08 02:55:07,412 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting d688efb702e9428980791db7888938c7, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=683, earliestPutTs=1680922505149 2023-04-08 02:55:07,417 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): d3e62a2534068dd21a6b490550cac850#C#compaction#352 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:07,825 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/7ea55a63987d4462b320a24be451fd37 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/7ea55a63987d4462b320a24be451fd37 2023-04-08 02:55:07,828 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in d3e62a2534068dd21a6b490550cac850/C of d3e62a2534068dd21a6b490550cac850 into 7ea55a63987d4462b320a24be451fd37(size=13.5 K), total size for store is 13.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:07,828 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:07,828 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850., storeName=d3e62a2534068dd21a6b490550cac850/C, priority=13, startTime=1680922506986; duration=0sec 2023-04-08 02:55:07,829 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:11,972 DEBUG [Thread-1626] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6919b2a0 to 127.0.0.1:64826 2023-04-08 02:55:11,972 DEBUG [Thread-1626] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:55:11,983 DEBUG [Thread-1624] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3cf5bdb7 to 127.0.0.1:64826 2023-04-08 02:55:11,983 DEBUG [Thread-1624] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:55:12,022 DEBUG [Thread-1620] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5c124820 to 127.0.0.1:64826 2023-04-08 02:55:12,022 DEBUG [Thread-1620] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:55:12,022 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-08 02:55:12,022 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 214 2023-04-08 02:55:12,022 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 44 2023-04-08 02:55:12,022 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 177 2023-04-08 02:55:12,022 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 52 2023-04-08 02:55:12,022 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 55 2023-04-08 02:55:12,022 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-08 02:55:12,022 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(398): read 6038 2023-04-08 02:55:12,023 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(398): read 6281 2023-04-08 02:55:12,023 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(398): read 5970 2023-04-08 02:55:12,023 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(398): read 6031 2023-04-08 02:55:12,023 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(398): read 6302 2023-04-08 02:55:12,023 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-08 02:55:12,023 INFO [Listener at localhost/39741] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-08 02:55:12,023 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x12e82edd to 127.0.0.1:64826 2023-04-08 02:55:12,023 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:55:12,023 INFO [Listener at localhost/39741] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-08 02:55:12,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-08 02:55:12,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure2.ProcedureExecutor(1029): Stored pid=31, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-08 02:55:12,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-08 02:55:12,027 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680922512027"}]},"ts":"1680922512027"} 2023-04-08 02:55:12,028 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-08 02:55:12,029 INFO [PEWorker-2] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-08 02:55:12,030 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d3e62a2534068dd21a6b490550cac850, UNASSIGN}] 2023-04-08 02:55:12,031 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=32, ppid=31, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=d3e62a2534068dd21a6b490550cac850, UNASSIGN 2023-04-08 02:55:12,032 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=d3e62a2534068dd21a6b490550cac850, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:12,032 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922512032"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680922512032"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680922512032"}]},"ts":"1680922512032"} 2023-04-08 02:55:12,033 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=33, ppid=32, state=RUNNABLE; CloseRegionProcedure d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983}] 2023-04-08 02:55:12,128 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-08 02:55:12,184 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close d3e62a2534068dd21a6b490550cac850 2023-04-08 02:55:12,185 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing d3e62a2534068dd21a6b490550cac850, disabling compactions & flushes 2023-04-08 02:55:12,185 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:12,185 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:12,185 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. after waiting 0 ms 2023-04-08 02:55:12,185 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:12,185 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing d3e62a2534068dd21a6b490550cac850 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-08 02:55:12,185 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=A 2023-04-08 02:55:12,185 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:12,185 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=B 2023-04-08 02:55:12,185 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:12,185 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK d3e62a2534068dd21a6b490550cac850, store=C 2023-04-08 02:55:12,185 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:12,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-08 02:55:12,591 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=693 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/8e6bcf2f4aea40a8b4a367f4595f97f9 2023-04-08 02:55:12,600 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=693 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/fc8c4cc76c8e49709296dc1c67a37a53 2023-04-08 02:55:12,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-08 02:55:13,010 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=693 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/bc9f5b0a13ef4a6fa7b5f2c886974e90 2023-04-08 02:55:13,013 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/A/8e6bcf2f4aea40a8b4a367f4595f97f9 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/8e6bcf2f4aea40a8b4a367f4595f97f9 2023-04-08 02:55:13,016 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/8e6bcf2f4aea40a8b4a367f4595f97f9, entries=100, sequenceid=693, filesize=9.5 K 2023-04-08 02:55:13,017 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/B/fc8c4cc76c8e49709296dc1c67a37a53 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/fc8c4cc76c8e49709296dc1c67a37a53 2023-04-08 02:55:13,020 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/fc8c4cc76c8e49709296dc1c67a37a53, entries=100, sequenceid=693, filesize=9.5 K 2023-04-08 02:55:13,020 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/.tmp/C/bc9f5b0a13ef4a6fa7b5f2c886974e90 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/bc9f5b0a13ef4a6fa7b5f2c886974e90 2023-04-08 02:55:13,023 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/bc9f5b0a13ef4a6fa7b5f2c886974e90, entries=100, sequenceid=693, filesize=9.5 K 2023-04-08 02:55:13,024 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=0 B/0 for d3e62a2534068dd21a6b490550cac850 in 839ms, sequenceid=693, compaction requested=false 2023-04-08 02:55:13,050 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/21df7a61da0d40f88f0158b0ed3f0876, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/7b358e84c4794107b9dac44972fba574, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/f24dbbad89a846e8989762e099350978, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/93c0f59b6b774e66b2d7ca0cd15f6e92, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/1f57ea055c0043b3b9b450ab6f386b40, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/3659dde192c243f19be7a4301bfba1a7, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/aee3fd6cbad14eceb604d3495bfc2f87, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/2dfed944a4174da8834bc2c5c3150785, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/423319047a3040c68e582e118623c814, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/e2418d2df4d44f5f86aa0805c657e8b6, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/7a256b999e9540a581c63623c8099904, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/ca1cc1c2640545e29c36a791ab2f91c8, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/31774130daf2498b93908415de5fc15f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/1162f92f7db44779bacdd68e5831c608, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/550ce511fbc747d59c977724737a9d3a, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/532fb819294f4cd1b6b75bd973ee406f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/ffb80df32ef2492eb73d3ed962897d41, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/2fbae948cce0453a995707498943d20c, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/8ae1e7a646614d41b7a478efa1e028b9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/a4235e0431d84c66923c515fde6ed748, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/081f5868ffb9469b96ce4c11b3ad27a6, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/cf400ef63737442588973f2dc11d7be9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/4a8ca4d0a9c5476d920418f9418b4f8e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/83223b976685460398248f86e8fd9d63, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/10c38b75155a45dba721e52823c9e635, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/871b98b8495048108c0e86861b1a49fa, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/459f2ee807d74a08898ce44af69bdf28, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/e42beb478658418aa0f86abbaca66f75, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/e270e87b4e454108804e9fd17003b1ac, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/2f1c303e29a14c18958c93e16a30b254, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/58cae7fbddb94205a0ec7368d5278df6, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/77606c9ad16e46a4b28652b408ebe5bc, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/69db4680af2c4b799dfaf2656aacbea1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/7123a3c634794da2a20b38c6754c212a, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/ae982203a8da425b8d9333063dc7ef8f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/a4c20daf19284992b2f78e4224508266, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/5effe2bd68f54744a2c86cc12cbe380f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/e898f1996830449ab32171a289f80645, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/ce9ac13585d84af1a17ac92f23862440, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/86f80b4295b84a7686944a21f681ede8, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/2f7011414c594a1f87355524b193a5a1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/6742d944eaae49868eb7fd69960dce9f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/775e58a24687413c8c7c0a8f6b7b091e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/0870d9d29e69485d8fe598d942c1a3d2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/9081ac96701242ac922110ea76bafa22, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/692f290bf50c402494cced7d7e14d30a, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/161b783e52ac45a5b6b88f930e75810c] to archive 2023-04-08 02:55:13,051 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 02:55:13,053 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/21df7a61da0d40f88f0158b0ed3f0876 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/21df7a61da0d40f88f0158b0ed3f0876 2023-04-08 02:55:13,055 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/7b358e84c4794107b9dac44972fba574 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/7b358e84c4794107b9dac44972fba574 2023-04-08 02:55:13,056 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/f24dbbad89a846e8989762e099350978 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/f24dbbad89a846e8989762e099350978 2023-04-08 02:55:13,057 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/93c0f59b6b774e66b2d7ca0cd15f6e92 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/93c0f59b6b774e66b2d7ca0cd15f6e92 2023-04-08 02:55:13,058 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/1f57ea055c0043b3b9b450ab6f386b40 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/1f57ea055c0043b3b9b450ab6f386b40 2023-04-08 02:55:13,059 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/3659dde192c243f19be7a4301bfba1a7 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/3659dde192c243f19be7a4301bfba1a7 2023-04-08 02:55:13,060 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/aee3fd6cbad14eceb604d3495bfc2f87 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/aee3fd6cbad14eceb604d3495bfc2f87 2023-04-08 02:55:13,061 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/2dfed944a4174da8834bc2c5c3150785 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/2dfed944a4174da8834bc2c5c3150785 2023-04-08 02:55:13,061 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/423319047a3040c68e582e118623c814 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/423319047a3040c68e582e118623c814 2023-04-08 02:55:13,062 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/e2418d2df4d44f5f86aa0805c657e8b6 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/e2418d2df4d44f5f86aa0805c657e8b6 2023-04-08 02:55:13,063 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/7a256b999e9540a581c63623c8099904 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/7a256b999e9540a581c63623c8099904 2023-04-08 02:55:13,064 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/ca1cc1c2640545e29c36a791ab2f91c8 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/ca1cc1c2640545e29c36a791ab2f91c8 2023-04-08 02:55:13,065 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/31774130daf2498b93908415de5fc15f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/31774130daf2498b93908415de5fc15f 2023-04-08 02:55:13,066 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/1162f92f7db44779bacdd68e5831c608 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/1162f92f7db44779bacdd68e5831c608 2023-04-08 02:55:13,066 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/550ce511fbc747d59c977724737a9d3a to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/550ce511fbc747d59c977724737a9d3a 2023-04-08 02:55:13,068 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/532fb819294f4cd1b6b75bd973ee406f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/532fb819294f4cd1b6b75bd973ee406f 2023-04-08 02:55:13,069 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/ffb80df32ef2492eb73d3ed962897d41 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/ffb80df32ef2492eb73d3ed962897d41 2023-04-08 02:55:13,069 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/2fbae948cce0453a995707498943d20c to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/2fbae948cce0453a995707498943d20c 2023-04-08 02:55:13,070 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/8ae1e7a646614d41b7a478efa1e028b9 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/8ae1e7a646614d41b7a478efa1e028b9 2023-04-08 02:55:13,071 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/a4235e0431d84c66923c515fde6ed748 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/a4235e0431d84c66923c515fde6ed748 2023-04-08 02:55:13,072 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/081f5868ffb9469b96ce4c11b3ad27a6 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/081f5868ffb9469b96ce4c11b3ad27a6 2023-04-08 02:55:13,073 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/cf400ef63737442588973f2dc11d7be9 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/cf400ef63737442588973f2dc11d7be9 2023-04-08 02:55:13,074 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/4a8ca4d0a9c5476d920418f9418b4f8e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/4a8ca4d0a9c5476d920418f9418b4f8e 2023-04-08 02:55:13,076 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/83223b976685460398248f86e8fd9d63 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/83223b976685460398248f86e8fd9d63 2023-04-08 02:55:13,077 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/10c38b75155a45dba721e52823c9e635 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/10c38b75155a45dba721e52823c9e635 2023-04-08 02:55:13,077 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/871b98b8495048108c0e86861b1a49fa to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/871b98b8495048108c0e86861b1a49fa 2023-04-08 02:55:13,078 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/459f2ee807d74a08898ce44af69bdf28 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/459f2ee807d74a08898ce44af69bdf28 2023-04-08 02:55:13,079 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/e42beb478658418aa0f86abbaca66f75 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/e42beb478658418aa0f86abbaca66f75 2023-04-08 02:55:13,080 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/e270e87b4e454108804e9fd17003b1ac to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/e270e87b4e454108804e9fd17003b1ac 2023-04-08 02:55:13,081 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/2f1c303e29a14c18958c93e16a30b254 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/2f1c303e29a14c18958c93e16a30b254 2023-04-08 02:55:13,082 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/58cae7fbddb94205a0ec7368d5278df6 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/58cae7fbddb94205a0ec7368d5278df6 2023-04-08 02:55:13,083 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/77606c9ad16e46a4b28652b408ebe5bc to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/77606c9ad16e46a4b28652b408ebe5bc 2023-04-08 02:55:13,084 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/69db4680af2c4b799dfaf2656aacbea1 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/69db4680af2c4b799dfaf2656aacbea1 2023-04-08 02:55:13,085 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/7123a3c634794da2a20b38c6754c212a to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/7123a3c634794da2a20b38c6754c212a 2023-04-08 02:55:13,086 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/ae982203a8da425b8d9333063dc7ef8f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/ae982203a8da425b8d9333063dc7ef8f 2023-04-08 02:55:13,087 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/a4c20daf19284992b2f78e4224508266 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/a4c20daf19284992b2f78e4224508266 2023-04-08 02:55:13,088 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/5effe2bd68f54744a2c86cc12cbe380f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/5effe2bd68f54744a2c86cc12cbe380f 2023-04-08 02:55:13,089 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/e898f1996830449ab32171a289f80645 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/e898f1996830449ab32171a289f80645 2023-04-08 02:55:13,089 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/ce9ac13585d84af1a17ac92f23862440 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/ce9ac13585d84af1a17ac92f23862440 2023-04-08 02:55:13,090 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/86f80b4295b84a7686944a21f681ede8 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/86f80b4295b84a7686944a21f681ede8 2023-04-08 02:55:13,091 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/2f7011414c594a1f87355524b193a5a1 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/2f7011414c594a1f87355524b193a5a1 2023-04-08 02:55:13,092 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/6742d944eaae49868eb7fd69960dce9f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/6742d944eaae49868eb7fd69960dce9f 2023-04-08 02:55:13,093 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/775e58a24687413c8c7c0a8f6b7b091e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/775e58a24687413c8c7c0a8f6b7b091e 2023-04-08 02:55:13,094 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/0870d9d29e69485d8fe598d942c1a3d2 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/0870d9d29e69485d8fe598d942c1a3d2 2023-04-08 02:55:13,095 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/9081ac96701242ac922110ea76bafa22 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/9081ac96701242ac922110ea76bafa22 2023-04-08 02:55:13,096 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/692f290bf50c402494cced7d7e14d30a to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/692f290bf50c402494cced7d7e14d30a 2023-04-08 02:55:13,097 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/161b783e52ac45a5b6b88f930e75810c to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/161b783e52ac45a5b6b88f930e75810c 2023-04-08 02:55:13,102 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/ec2af5492feb49128a938cc0c35d349c, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/4a0acab50073490ebf42e95ac38e6c09, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e4ddbf02c5454155a0a7f145ff80683d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e834842d1dbd496a971e44156f3a3f75, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/8feaee6405074216a582941bcc279d23, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/da1228e4bb1f4aa48345db8b302ffa3f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/db8f8a73d85f4cd1b5fe235697d24d58, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d9e23cab6e4c461a9db0439057b0d869, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/b80eaaed15b04bc4bc5946ddae8f2265, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/51ba9f71b0eb4b71b7fad023bdc60caa, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/2538b16666604a7a9b9b5ead38b6acec, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/f54b65712bda43bdb983609884a3a536, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/95cb51ce57f24bbe86e4c6aef9e1f42f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/2e4118c9fcb1494d8b29d59040b6255a, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/c2fc0a4d1c4c442984d96d85703a9119, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/9645eea91b3f4654a456645adca380be, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d848cf5774054fbc9593b3124c7f705b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/aa9e7cedcdf843288ed7827bc12bdc56, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/8b8054e87cf547d597aba62d21354a6d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e4eb91ade5ee4e52bbf1a699c7162103, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/ad50f33edc224fca94917315d4681bb1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/06eaf40d3f234b85b12a8342cab4a925, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/2500fd73318b401dab88b03f7ef73e1d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d2392b9f910d4d619c052a392506e837, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/98cd3f7a91d74f1a827f621c9712504e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/a42a4c94299d4aa2964634fbb1bb42dc, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/6e5a84ecf2e6404d92b5ea40501e6813, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e6584fd4292743ed8e8c8629585481e4, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/f52fd04a4e9f450481473ffb277b384f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e0e09aa2d81b47ac8c9fed792c22ac8f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/8e38f255f7ae486e8df976c3cf4993a8, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d614477ba87b4837855e8c38f336576e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/c8e40f4bfbd74e23b8c86301d1a28792, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/65afa1e381f04a04ba887afd9f533488, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/600c035254634338838bf4601ba3bd28, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/ec5c11b9f14441d5ab5a9b71f185b4fc, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/39322bf082634a479465bdac26ec9df6, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/8159382b9f5b4c3196b652c3a781af89, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/3a39b47b08774b60874ff633656440f0, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d1f5ee65227d4855953ecd8a4a854aeb, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/25e497722d1c42e5b70ea3511dd7f5f2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/ade5ddc28d094489884c062305cb7eff, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/afe7fafe57d94ed888002440028059e8, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/27774f5acf3a42c09467f19e3498b034, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/341bad5545e143e298e5baeccde2d88d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/37527ab0dfdf459aa682fcc1089ba805, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/bf27f2efdcd7449795a88556d5be8b88] to archive 2023-04-08 02:55:13,103 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 02:55:13,104 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/ec2af5492feb49128a938cc0c35d349c to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/ec2af5492feb49128a938cc0c35d349c 2023-04-08 02:55:13,105 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/4a0acab50073490ebf42e95ac38e6c09 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/4a0acab50073490ebf42e95ac38e6c09 2023-04-08 02:55:13,106 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e4ddbf02c5454155a0a7f145ff80683d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e4ddbf02c5454155a0a7f145ff80683d 2023-04-08 02:55:13,107 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e834842d1dbd496a971e44156f3a3f75 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e834842d1dbd496a971e44156f3a3f75 2023-04-08 02:55:13,108 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/8feaee6405074216a582941bcc279d23 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/8feaee6405074216a582941bcc279d23 2023-04-08 02:55:13,108 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/da1228e4bb1f4aa48345db8b302ffa3f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/da1228e4bb1f4aa48345db8b302ffa3f 2023-04-08 02:55:13,109 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/db8f8a73d85f4cd1b5fe235697d24d58 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/db8f8a73d85f4cd1b5fe235697d24d58 2023-04-08 02:55:13,110 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d9e23cab6e4c461a9db0439057b0d869 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d9e23cab6e4c461a9db0439057b0d869 2023-04-08 02:55:13,111 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/b80eaaed15b04bc4bc5946ddae8f2265 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/b80eaaed15b04bc4bc5946ddae8f2265 2023-04-08 02:55:13,111 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/51ba9f71b0eb4b71b7fad023bdc60caa to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/51ba9f71b0eb4b71b7fad023bdc60caa 2023-04-08 02:55:13,112 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/2538b16666604a7a9b9b5ead38b6acec to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/2538b16666604a7a9b9b5ead38b6acec 2023-04-08 02:55:13,113 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/f54b65712bda43bdb983609884a3a536 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/f54b65712bda43bdb983609884a3a536 2023-04-08 02:55:13,114 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/95cb51ce57f24bbe86e4c6aef9e1f42f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/95cb51ce57f24bbe86e4c6aef9e1f42f 2023-04-08 02:55:13,114 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/2e4118c9fcb1494d8b29d59040b6255a to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/2e4118c9fcb1494d8b29d59040b6255a 2023-04-08 02:55:13,115 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/c2fc0a4d1c4c442984d96d85703a9119 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/c2fc0a4d1c4c442984d96d85703a9119 2023-04-08 02:55:13,116 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/9645eea91b3f4654a456645adca380be to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/9645eea91b3f4654a456645adca380be 2023-04-08 02:55:13,117 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d848cf5774054fbc9593b3124c7f705b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d848cf5774054fbc9593b3124c7f705b 2023-04-08 02:55:13,117 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/aa9e7cedcdf843288ed7827bc12bdc56 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/aa9e7cedcdf843288ed7827bc12bdc56 2023-04-08 02:55:13,118 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/8b8054e87cf547d597aba62d21354a6d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/8b8054e87cf547d597aba62d21354a6d 2023-04-08 02:55:13,119 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e4eb91ade5ee4e52bbf1a699c7162103 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e4eb91ade5ee4e52bbf1a699c7162103 2023-04-08 02:55:13,120 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/ad50f33edc224fca94917315d4681bb1 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/ad50f33edc224fca94917315d4681bb1 2023-04-08 02:55:13,121 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/06eaf40d3f234b85b12a8342cab4a925 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/06eaf40d3f234b85b12a8342cab4a925 2023-04-08 02:55:13,121 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/2500fd73318b401dab88b03f7ef73e1d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/2500fd73318b401dab88b03f7ef73e1d 2023-04-08 02:55:13,122 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d2392b9f910d4d619c052a392506e837 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d2392b9f910d4d619c052a392506e837 2023-04-08 02:55:13,123 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/98cd3f7a91d74f1a827f621c9712504e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/98cd3f7a91d74f1a827f621c9712504e 2023-04-08 02:55:13,124 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/a42a4c94299d4aa2964634fbb1bb42dc to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/a42a4c94299d4aa2964634fbb1bb42dc 2023-04-08 02:55:13,124 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/6e5a84ecf2e6404d92b5ea40501e6813 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/6e5a84ecf2e6404d92b5ea40501e6813 2023-04-08 02:55:13,125 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e6584fd4292743ed8e8c8629585481e4 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e6584fd4292743ed8e8c8629585481e4 2023-04-08 02:55:13,126 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/f52fd04a4e9f450481473ffb277b384f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/f52fd04a4e9f450481473ffb277b384f 2023-04-08 02:55:13,127 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e0e09aa2d81b47ac8c9fed792c22ac8f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/e0e09aa2d81b47ac8c9fed792c22ac8f 2023-04-08 02:55:13,127 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/8e38f255f7ae486e8df976c3cf4993a8 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/8e38f255f7ae486e8df976c3cf4993a8 2023-04-08 02:55:13,128 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d614477ba87b4837855e8c38f336576e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d614477ba87b4837855e8c38f336576e 2023-04-08 02:55:13,129 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/c8e40f4bfbd74e23b8c86301d1a28792 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/c8e40f4bfbd74e23b8c86301d1a28792 2023-04-08 02:55:13,130 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/65afa1e381f04a04ba887afd9f533488 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/65afa1e381f04a04ba887afd9f533488 2023-04-08 02:55:13,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-08 02:55:13,131 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/600c035254634338838bf4601ba3bd28 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/600c035254634338838bf4601ba3bd28 2023-04-08 02:55:13,131 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/ec5c11b9f14441d5ab5a9b71f185b4fc to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/ec5c11b9f14441d5ab5a9b71f185b4fc 2023-04-08 02:55:13,132 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/39322bf082634a479465bdac26ec9df6 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/39322bf082634a479465bdac26ec9df6 2023-04-08 02:55:13,133 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/8159382b9f5b4c3196b652c3a781af89 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/8159382b9f5b4c3196b652c3a781af89 2023-04-08 02:55:13,134 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/3a39b47b08774b60874ff633656440f0 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/3a39b47b08774b60874ff633656440f0 2023-04-08 02:55:13,134 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d1f5ee65227d4855953ecd8a4a854aeb to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/d1f5ee65227d4855953ecd8a4a854aeb 2023-04-08 02:55:13,135 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/25e497722d1c42e5b70ea3511dd7f5f2 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/25e497722d1c42e5b70ea3511dd7f5f2 2023-04-08 02:55:13,136 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/ade5ddc28d094489884c062305cb7eff to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/ade5ddc28d094489884c062305cb7eff 2023-04-08 02:55:13,137 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/afe7fafe57d94ed888002440028059e8 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/afe7fafe57d94ed888002440028059e8 2023-04-08 02:55:13,138 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/27774f5acf3a42c09467f19e3498b034 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/27774f5acf3a42c09467f19e3498b034 2023-04-08 02:55:13,138 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/341bad5545e143e298e5baeccde2d88d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/341bad5545e143e298e5baeccde2d88d 2023-04-08 02:55:13,139 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/37527ab0dfdf459aa682fcc1089ba805 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/37527ab0dfdf459aa682fcc1089ba805 2023-04-08 02:55:13,140 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/bf27f2efdcd7449795a88556d5be8b88 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/bf27f2efdcd7449795a88556d5be8b88 2023-04-08 02:55:13,143 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/23c0cb46ada34eb1a3790c429cecdd98, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/17f0a683716249f98b90f6c2bfac2063, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/e8555f4db1b541c39bc6c752959b3dbb, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/eafa802837c84e58acc988961514407d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/2de979b47e894794a929205856c44c12, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/29efa88951b74ba18a0de24371408fa0, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/7cb603f1b1704b5cacf1955aee6a6426, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/417c08dd7e434b26aa3e37b3d2b423ce, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/7040a0f67a0d414e845399f126ad5c31, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/48379221b677477bb49c122e4a792eb2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/1a5377a20f294ac988dd6c5f60beccf0, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/b9d6edbc704c4e66b7abc835d4a06502, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/bb522fe7018b4d419c11527865976620, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/b2bfc5602d0a4a508049209cfc218bb0, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/65f8e93f1003412da014622ab5513830, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/a2c687c7a1f8419380ab5c91b7b5f6b8, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/00e350c3b5614847bbcbf120a93bb663, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/98f2a05a2a78439ba93562cc4dc4b5b0, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/c289668a5f0841bdade9fb51511e19c4, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/aa98a0988bab4340a44f52761f2d2b62, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/2852ab4ca852432ebb55d93e507b02cc, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/36c9f184e80944d9ac3f590dd4978e1f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/f635ca9c9f7e4f488eda116355d196ad, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/1e0c45baffc74543b9373ea0a6481e57, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/34aa33d84ba649b18a9991a8c37937c1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/78b5b3fc994b496fa57ee4a9de040e19, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/336d78fba99943299b5d757721c8b5ac, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/4375c3dcfce64b33be21d1694fab29be, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/6045515eb7e346e6ae7336b8ff5d34ea, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/8fb05f56362744b1af3ef8f1d14e150d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/3a0b16f6d510451a8d1a150ff35de2c8, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/11d926fd80854bcf8bd50c9419ad032d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/c0b5e09ab75544c2819c463cabc76a75, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/d12e7e6f606a4a70892bcc65bfdf1bbf, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/e7fc26e2c2d04f1bbd47893e82d943aa, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/f7167b5985104042963f2f27701c8c71, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/651757c81ba94e7ab7554ea4a3688447, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/7d800ad830ee4e8889ad407b1b6f6ad1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/d743cc053e1442e6aa2f4a75916ae62a, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/3f5b07e818e442b2953b4375da32a024, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/77cad5ab77b3478db8c6a18972166e02, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/0b3d61a800c84e60914595f87ff27192, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/a258fb818fb84050805adb9555bbe966, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/64e83a5d70ba49f38492b3bdeefd6239, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/62185f6e01e94ab49f528f76fca40362, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/09395a24ac11431997f2ccdd58fd6e97, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/d688efb702e9428980791db7888938c7] to archive 2023-04-08 02:55:13,144 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 02:55:13,145 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/23c0cb46ada34eb1a3790c429cecdd98 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/23c0cb46ada34eb1a3790c429cecdd98 2023-04-08 02:55:13,146 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/17f0a683716249f98b90f6c2bfac2063 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/17f0a683716249f98b90f6c2bfac2063 2023-04-08 02:55:13,147 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/e8555f4db1b541c39bc6c752959b3dbb to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/e8555f4db1b541c39bc6c752959b3dbb 2023-04-08 02:55:13,147 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/eafa802837c84e58acc988961514407d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/eafa802837c84e58acc988961514407d 2023-04-08 02:55:13,148 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/2de979b47e894794a929205856c44c12 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/2de979b47e894794a929205856c44c12 2023-04-08 02:55:13,149 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/29efa88951b74ba18a0de24371408fa0 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/29efa88951b74ba18a0de24371408fa0 2023-04-08 02:55:13,150 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/7cb603f1b1704b5cacf1955aee6a6426 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/7cb603f1b1704b5cacf1955aee6a6426 2023-04-08 02:55:13,151 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/417c08dd7e434b26aa3e37b3d2b423ce to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/417c08dd7e434b26aa3e37b3d2b423ce 2023-04-08 02:55:13,152 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/7040a0f67a0d414e845399f126ad5c31 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/7040a0f67a0d414e845399f126ad5c31 2023-04-08 02:55:13,152 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/48379221b677477bb49c122e4a792eb2 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/48379221b677477bb49c122e4a792eb2 2023-04-08 02:55:13,153 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/1a5377a20f294ac988dd6c5f60beccf0 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/1a5377a20f294ac988dd6c5f60beccf0 2023-04-08 02:55:13,154 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/b9d6edbc704c4e66b7abc835d4a06502 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/b9d6edbc704c4e66b7abc835d4a06502 2023-04-08 02:55:13,155 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/bb522fe7018b4d419c11527865976620 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/bb522fe7018b4d419c11527865976620 2023-04-08 02:55:13,156 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/b2bfc5602d0a4a508049209cfc218bb0 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/b2bfc5602d0a4a508049209cfc218bb0 2023-04-08 02:55:13,157 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/65f8e93f1003412da014622ab5513830 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/65f8e93f1003412da014622ab5513830 2023-04-08 02:55:13,157 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/a2c687c7a1f8419380ab5c91b7b5f6b8 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/a2c687c7a1f8419380ab5c91b7b5f6b8 2023-04-08 02:55:13,158 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/00e350c3b5614847bbcbf120a93bb663 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/00e350c3b5614847bbcbf120a93bb663 2023-04-08 02:55:13,159 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/98f2a05a2a78439ba93562cc4dc4b5b0 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/98f2a05a2a78439ba93562cc4dc4b5b0 2023-04-08 02:55:13,160 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/c289668a5f0841bdade9fb51511e19c4 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/c289668a5f0841bdade9fb51511e19c4 2023-04-08 02:55:13,161 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/aa98a0988bab4340a44f52761f2d2b62 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/aa98a0988bab4340a44f52761f2d2b62 2023-04-08 02:55:13,161 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/2852ab4ca852432ebb55d93e507b02cc to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/2852ab4ca852432ebb55d93e507b02cc 2023-04-08 02:55:13,162 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/36c9f184e80944d9ac3f590dd4978e1f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/36c9f184e80944d9ac3f590dd4978e1f 2023-04-08 02:55:13,163 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/f635ca9c9f7e4f488eda116355d196ad to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/f635ca9c9f7e4f488eda116355d196ad 2023-04-08 02:55:13,164 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/1e0c45baffc74543b9373ea0a6481e57 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/1e0c45baffc74543b9373ea0a6481e57 2023-04-08 02:55:13,165 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/34aa33d84ba649b18a9991a8c37937c1 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/34aa33d84ba649b18a9991a8c37937c1 2023-04-08 02:55:13,165 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/78b5b3fc994b496fa57ee4a9de040e19 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/78b5b3fc994b496fa57ee4a9de040e19 2023-04-08 02:55:13,166 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/336d78fba99943299b5d757721c8b5ac to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/336d78fba99943299b5d757721c8b5ac 2023-04-08 02:55:13,167 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/4375c3dcfce64b33be21d1694fab29be to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/4375c3dcfce64b33be21d1694fab29be 2023-04-08 02:55:13,168 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/6045515eb7e346e6ae7336b8ff5d34ea to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/6045515eb7e346e6ae7336b8ff5d34ea 2023-04-08 02:55:13,169 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/8fb05f56362744b1af3ef8f1d14e150d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/8fb05f56362744b1af3ef8f1d14e150d 2023-04-08 02:55:13,170 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/3a0b16f6d510451a8d1a150ff35de2c8 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/3a0b16f6d510451a8d1a150ff35de2c8 2023-04-08 02:55:13,171 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/11d926fd80854bcf8bd50c9419ad032d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/11d926fd80854bcf8bd50c9419ad032d 2023-04-08 02:55:13,171 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/c0b5e09ab75544c2819c463cabc76a75 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/c0b5e09ab75544c2819c463cabc76a75 2023-04-08 02:55:13,172 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/d12e7e6f606a4a70892bcc65bfdf1bbf to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/d12e7e6f606a4a70892bcc65bfdf1bbf 2023-04-08 02:55:13,173 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/e7fc26e2c2d04f1bbd47893e82d943aa to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/e7fc26e2c2d04f1bbd47893e82d943aa 2023-04-08 02:55:13,174 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/f7167b5985104042963f2f27701c8c71 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/f7167b5985104042963f2f27701c8c71 2023-04-08 02:55:13,175 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/651757c81ba94e7ab7554ea4a3688447 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/651757c81ba94e7ab7554ea4a3688447 2023-04-08 02:55:13,175 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/7d800ad830ee4e8889ad407b1b6f6ad1 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/7d800ad830ee4e8889ad407b1b6f6ad1 2023-04-08 02:55:13,176 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/d743cc053e1442e6aa2f4a75916ae62a to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/d743cc053e1442e6aa2f4a75916ae62a 2023-04-08 02:55:13,177 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/3f5b07e818e442b2953b4375da32a024 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/3f5b07e818e442b2953b4375da32a024 2023-04-08 02:55:13,178 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/77cad5ab77b3478db8c6a18972166e02 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/77cad5ab77b3478db8c6a18972166e02 2023-04-08 02:55:13,178 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/0b3d61a800c84e60914595f87ff27192 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/0b3d61a800c84e60914595f87ff27192 2023-04-08 02:55:13,179 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/a258fb818fb84050805adb9555bbe966 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/a258fb818fb84050805adb9555bbe966 2023-04-08 02:55:13,180 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/64e83a5d70ba49f38492b3bdeefd6239 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/64e83a5d70ba49f38492b3bdeefd6239 2023-04-08 02:55:13,181 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/62185f6e01e94ab49f528f76fca40362 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/62185f6e01e94ab49f528f76fca40362 2023-04-08 02:55:13,181 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/09395a24ac11431997f2ccdd58fd6e97 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/09395a24ac11431997f2ccdd58fd6e97 2023-04-08 02:55:13,182 DEBUG [StoreCloser-TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/d688efb702e9428980791db7888938c7 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/d688efb702e9428980791db7888938c7 2023-04-08 02:55:13,186 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/recovered.edits/696.seqid, newMaxSeqId=696, maxSeqId=1 2023-04-08 02:55:13,187 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850. 2023-04-08 02:55:13,187 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for d3e62a2534068dd21a6b490550cac850: 2023-04-08 02:55:13,189 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed d3e62a2534068dd21a6b490550cac850 2023-04-08 02:55:13,189 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=32 updating hbase:meta row=d3e62a2534068dd21a6b490550cac850, regionState=CLOSED 2023-04-08 02:55:13,189 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922513189"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680922513189"}]},"ts":"1680922513189"} 2023-04-08 02:55:13,191 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=33, resume processing ppid=32 2023-04-08 02:55:13,191 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=33, ppid=32, state=SUCCESS; CloseRegionProcedure d3e62a2534068dd21a6b490550cac850, server=jenkins-hbase4.apache.org,35089,1680922417983 in 1.1570 sec 2023-04-08 02:55:13,192 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=32, resume processing ppid=31 2023-04-08 02:55:13,192 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=32, ppid=31, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=d3e62a2534068dd21a6b490550cac850, UNASSIGN in 1.1610 sec 2023-04-08 02:55:13,192 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680922513192"}]},"ts":"1680922513192"} 2023-04-08 02:55:13,193 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-08 02:55:13,194 INFO [PEWorker-2] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-08 02:55:13,195 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=31, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.1710 sec 2023-04-08 02:55:14,131 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=31 2023-04-08 02:55:14,131 INFO [Listener at localhost/39741] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 31 completed 2023-04-08 02:55:14,132 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-08 02:55:14,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure2.ProcedureExecutor(1029): Stored pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:55:14,134 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=34, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:55:14,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-08 02:55:14,134 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=34, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:55:14,138 DEBUG [HFileArchiver-7] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850 2023-04-08 02:55:14,140 DEBUG [HFileArchiver-7] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A, FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B, FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C, FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/recovered.edits] 2023-04-08 02:55:14,143 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/8e6bcf2f4aea40a8b4a367f4595f97f9 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/8e6bcf2f4aea40a8b4a367f4595f97f9 2023-04-08 02:55:14,144 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/fbd98bf1d3a04733bf8f07f01aec24d1 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/A/fbd98bf1d3a04733bf8f07f01aec24d1 2023-04-08 02:55:14,147 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/8b1e229ee4c34594ac27eae2dad4bcea to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/8b1e229ee4c34594ac27eae2dad4bcea 2023-04-08 02:55:14,148 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/fc8c4cc76c8e49709296dc1c67a37a53 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/B/fc8c4cc76c8e49709296dc1c67a37a53 2023-04-08 02:55:14,150 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/7ea55a63987d4462b320a24be451fd37 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/7ea55a63987d4462b320a24be451fd37 2023-04-08 02:55:14,151 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/bc9f5b0a13ef4a6fa7b5f2c886974e90 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/C/bc9f5b0a13ef4a6fa7b5f2c886974e90 2023-04-08 02:55:14,154 DEBUG [HFileArchiver-7] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/recovered.edits/696.seqid to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850/recovered.edits/696.seqid 2023-04-08 02:55:14,154 DEBUG [HFileArchiver-7] backup.HFileArchiver(596): Deleted hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/d3e62a2534068dd21a6b490550cac850 2023-04-08 02:55:14,154 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-08 02:55:14,156 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=34, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:55:14,157 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-08 02:55:14,159 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-08 02:55:14,160 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=34, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:55:14,160 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-08 02:55:14,160 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680922514160"}]},"ts":"9223372036854775807"} 2023-04-08 02:55:14,162 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-08 02:55:14,162 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => d3e62a2534068dd21a6b490550cac850, NAME => 'TestAcidGuarantees,,1680922484524.d3e62a2534068dd21a6b490550cac850.', STARTKEY => '', ENDKEY => ''}] 2023-04-08 02:55:14,162 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-08 02:55:14,162 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680922514162"}]},"ts":"9223372036854775807"} 2023-04-08 02:55:14,163 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-08 02:55:14,165 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=34, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:55:14,166 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=34, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 33 msec 2023-04-08 02:55:14,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=34 2023-04-08 02:55:14,235 INFO [Listener at localhost/39741] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 34 completed 2023-04-08 02:55:14,247 INFO [Listener at localhost/39741] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testGetAtomicity Thread=225 (was 224) - Thread LEAK? -, OpenFileDescriptor=579 (was 534) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=401 (was 363) - SystemLoadAverage LEAK? -, ProcessCount=170 (was 170), AvailableMemoryMB=1466 (was 1633) 2023-04-08 02:55:14,258 INFO [Listener at localhost/39741] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=225, OpenFileDescriptor=579, MaxFileDescriptor=60000, SystemLoadAverage=401, ProcessCount=170, AvailableMemoryMB=1465 2023-04-08 02:55:14,259 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-08 02:55:14,260 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-08 02:55:14,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure2.ProcedureExecutor(1029): Stored pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-08 02:55:14,262 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-08 02:55:14,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 35 2023-04-08 02:55:14,262 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-08 02:55:14,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-08 02:55:14,263 DEBUG [HFileArchiver-8] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:14,264 DEBUG [HFileArchiver-8] backup.HFileArchiver(153): Directory hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b empty. 2023-04-08 02:55:14,264 DEBUG [HFileArchiver-8] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:14,264 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-08 02:55:14,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-08 02:55:14,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-08 02:55:14,671 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-08 02:55:14,672 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 8be8843d5bbff3d4194de143513dc64b, NAME => 'TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp 2023-04-08 02:55:14,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-08 02:55:15,078 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 02:55:15,078 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 8be8843d5bbff3d4194de143513dc64b, disabling compactions & flushes 2023-04-08 02:55:15,078 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:15,078 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:15,078 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. after waiting 0 ms 2023-04-08 02:55:15,078 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:15,078 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:15,078 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:15,080 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-08 02:55:15,081 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922515081"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680922515081"}]},"ts":"1680922515081"} 2023-04-08 02:55:15,082 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-08 02:55:15,082 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-08 02:55:15,083 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680922515082"}]},"ts":"1680922515082"} 2023-04-08 02:55:15,083 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-08 02:55:15,087 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8be8843d5bbff3d4194de143513dc64b, ASSIGN}] 2023-04-08 02:55:15,088 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8be8843d5bbff3d4194de143513dc64b, ASSIGN 2023-04-08 02:55:15,089 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=36, ppid=35, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=8be8843d5bbff3d4194de143513dc64b, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35089,1680922417983; forceNewPlan=false, retain=false 2023-04-08 02:55:15,240 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=8be8843d5bbff3d4194de143513dc64b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:15,240 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922515240"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680922515240"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680922515240"}]},"ts":"1680922515240"} 2023-04-08 02:55:15,241 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=37, ppid=36, state=RUNNABLE; OpenRegionProcedure 8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983}] 2023-04-08 02:55:15,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-08 02:55:15,396 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:15,396 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8be8843d5bbff3d4194de143513dc64b, NAME => 'TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.', STARTKEY => '', ENDKEY => ''} 2023-04-08 02:55:15,396 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:15,396 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 02:55:15,396 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:15,396 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:15,397 INFO [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:15,399 DEBUG [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A 2023-04-08 02:55:15,399 DEBUG [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A 2023-04-08 02:55:15,399 INFO [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 02:55:15,399 INFO [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8be8843d5bbff3d4194de143513dc64b columnFamilyName A 2023-04-08 02:55:15,399 INFO [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] regionserver.HStore(310): Store=8be8843d5bbff3d4194de143513dc64b/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:55:15,399 INFO [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:15,400 DEBUG [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B 2023-04-08 02:55:15,400 DEBUG [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B 2023-04-08 02:55:15,401 INFO [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 02:55:15,401 INFO [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8be8843d5bbff3d4194de143513dc64b columnFamilyName B 2023-04-08 02:55:15,401 INFO [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] regionserver.HStore(310): Store=8be8843d5bbff3d4194de143513dc64b/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:55:15,401 INFO [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:15,402 DEBUG [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C 2023-04-08 02:55:15,402 DEBUG [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C 2023-04-08 02:55:15,402 INFO [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 02:55:15,402 INFO [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8be8843d5bbff3d4194de143513dc64b columnFamilyName C 2023-04-08 02:55:15,403 INFO [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] regionserver.HStore(310): Store=8be8843d5bbff3d4194de143513dc64b/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:55:15,403 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:15,403 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:15,404 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:15,405 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-08 02:55:15,406 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:15,407 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-08 02:55:15,408 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 8be8843d5bbff3d4194de143513dc64b; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=65485296, jitterRate=-0.02419304847717285}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-08 02:55:15,408 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:15,409 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., pid=37, masterSystemTime=1680922515393 2023-04-08 02:55:15,410 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:15,410 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:15,410 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=36 updating hbase:meta row=8be8843d5bbff3d4194de143513dc64b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:15,410 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922515410"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680922515410"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680922515410"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680922515410"}]},"ts":"1680922515410"} 2023-04-08 02:55:15,412 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=37, resume processing ppid=36 2023-04-08 02:55:15,412 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=37, ppid=36, state=SUCCESS; OpenRegionProcedure 8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 in 170 msec 2023-04-08 02:55:15,413 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=36, resume processing ppid=35 2023-04-08 02:55:15,413 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=36, ppid=35, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8be8843d5bbff3d4194de143513dc64b, ASSIGN in 325 msec 2023-04-08 02:55:15,414 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-08 02:55:15,414 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680922515414"}]},"ts":"1680922515414"} 2023-04-08 02:55:15,415 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-08 02:55:15,416 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=35, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-08 02:55:15,417 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=35, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1560 sec 2023-04-08 02:55:16,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=35 2023-04-08 02:55:16,367 INFO [Listener at localhost/39741] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 35 completed 2023-04-08 02:55:16,368 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x0fd3d54e to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:55:16,373 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6434ce29, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:55:16,374 DEBUG [Listener at localhost/39741] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:55:16,376 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47698, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:55:16,377 DEBUG [Listener at localhost/39741] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-08 02:55:16,378 INFO [RS-EventLoopGroup-1-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33018, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-08 02:55:16,379 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-08 02:55:16,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-08 02:55:16,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure2.ProcedureExecutor(1029): Stored pid=38, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-08 02:55:16,791 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-08 02:55:16,792 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-08 02:55:16,792 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-08 02:55:16,794 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=39, ppid=38, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-08 02:55:16,796 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8be8843d5bbff3d4194de143513dc64b, REOPEN/MOVE}] 2023-04-08 02:55:16,797 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8be8843d5bbff3d4194de143513dc64b, REOPEN/MOVE 2023-04-08 02:55:16,797 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=8be8843d5bbff3d4194de143513dc64b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:16,797 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922516797"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680922516797"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680922516797"}]},"ts":"1680922516797"} 2023-04-08 02:55:16,798 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=41, ppid=40, state=RUNNABLE; CloseRegionProcedure 8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983}] 2023-04-08 02:55:16,950 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:16,951 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 8be8843d5bbff3d4194de143513dc64b, disabling compactions & flushes 2023-04-08 02:55:16,951 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:16,951 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:16,951 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. after waiting 0 ms 2023-04-08 02:55:16,951 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:16,954 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-08 02:55:16,954 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:16,954 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:16,954 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 8be8843d5bbff3d4194de143513dc64b to self. 2023-04-08 02:55:16,955 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:16,956 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=8be8843d5bbff3d4194de143513dc64b, regionState=CLOSED 2023-04-08 02:55:16,956 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922516956"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680922516956"}]},"ts":"1680922516956"} 2023-04-08 02:55:16,958 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=41, resume processing ppid=40 2023-04-08 02:55:16,958 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=41, ppid=40, state=SUCCESS; CloseRegionProcedure 8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 in 159 msec 2023-04-08 02:55:16,958 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=40, ppid=39, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=8be8843d5bbff3d4194de143513dc64b, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,35089,1680922417983; forceNewPlan=false, retain=true 2023-04-08 02:55:17,109 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=8be8843d5bbff3d4194de143513dc64b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,109 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922517108"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680922517108"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680922517108"}]},"ts":"1680922517108"} 2023-04-08 02:55:17,110 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=42, ppid=40, state=RUNNABLE; OpenRegionProcedure 8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983}] 2023-04-08 02:55:17,264 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:17,264 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 8be8843d5bbff3d4194de143513dc64b, NAME => 'TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.', STARTKEY => '', ENDKEY => ''} 2023-04-08 02:55:17,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:17,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 02:55:17,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:17,265 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:17,266 INFO [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:17,267 DEBUG [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A 2023-04-08 02:55:17,267 DEBUG [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A 2023-04-08 02:55:17,267 INFO [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 02:55:17,268 INFO [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8be8843d5bbff3d4194de143513dc64b columnFamilyName A 2023-04-08 02:55:17,269 INFO [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] regionserver.HStore(310): Store=8be8843d5bbff3d4194de143513dc64b/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:55:17,269 INFO [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:17,269 DEBUG [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B 2023-04-08 02:55:17,270 DEBUG [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B 2023-04-08 02:55:17,270 INFO [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 02:55:17,270 INFO [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8be8843d5bbff3d4194de143513dc64b columnFamilyName B 2023-04-08 02:55:17,270 INFO [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] regionserver.HStore(310): Store=8be8843d5bbff3d4194de143513dc64b/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:55:17,270 INFO [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:17,271 DEBUG [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C 2023-04-08 02:55:17,271 DEBUG [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C 2023-04-08 02:55:17,271 INFO [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 02:55:17,271 INFO [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 8be8843d5bbff3d4194de143513dc64b columnFamilyName C 2023-04-08 02:55:17,272 INFO [StoreOpener-8be8843d5bbff3d4194de143513dc64b-1] regionserver.HStore(310): Store=8be8843d5bbff3d4194de143513dc64b/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:55:17,272 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:17,272 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:17,273 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:17,275 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-08 02:55:17,276 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:17,277 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 8be8843d5bbff3d4194de143513dc64b; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=69727703, jitterRate=0.0390237420797348}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-08 02:55:17,277 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:17,277 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., pid=42, masterSystemTime=1680922517261 2023-04-08 02:55:17,278 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:17,278 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:17,279 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=40 updating hbase:meta row=8be8843d5bbff3d4194de143513dc64b, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,279 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922517279"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680922517279"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680922517279"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680922517279"}]},"ts":"1680922517279"} 2023-04-08 02:55:17,280 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=42, resume processing ppid=40 2023-04-08 02:55:17,280 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=42, ppid=40, state=SUCCESS; OpenRegionProcedure 8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 in 170 msec 2023-04-08 02:55:17,282 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=40, resume processing ppid=39 2023-04-08 02:55:17,282 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=40, ppid=39, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8be8843d5bbff3d4194de143513dc64b, REOPEN/MOVE in 485 msec 2023-04-08 02:55:17,283 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=39, resume processing ppid=38 2023-04-08 02:55:17,283 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=39, ppid=38, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 488 msec 2023-04-08 02:55:17,284 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=38, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 904 msec 2023-04-08 02:55:17,285 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=38 2023-04-08 02:55:17,286 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x1a5ab074 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:55:17,293 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@bb27fd4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:55:17,294 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x0bcfc1e0 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:55:17,297 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@445919fd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:55:17,298 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x38341134 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:55:17,301 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4d010713, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:55:17,301 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x71b5058b to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:55:17,305 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@69100738, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:55:17,305 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x3620ee35 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:55:17,315 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f603c27, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:55:17,316 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x7e01d5e5 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:55:17,319 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@10ee5f06, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:55:17,319 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x51ccd8b7 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:55:17,323 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@21128196, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:55:17,323 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x68a0e21a to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:55:17,328 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f9e86d9, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:55:17,328 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x1ec12b02 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:55:17,333 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@711ebaf2, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:55:17,333 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x1d7ba242 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:55:17,336 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@a7503d4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:55:17,342 DEBUG [hconnection-0x4b5ef87c-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:55:17,342 DEBUG [hconnection-0x46417b61-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:55:17,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:17,342 DEBUG [hconnection-0x32df1303-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:55:17,343 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47700, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:55:17,344 DEBUG [hconnection-0x1fc2fdea-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:55:17,345 DEBUG [hconnection-0x38d6812d-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:55:17,345 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47714, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:55:17,345 DEBUG [hconnection-0x51304042-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:55:17,345 DEBUG [hconnection-0x69225fe9-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:55:17,345 DEBUG [hconnection-0x3b0b92a1-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:55:17,352 DEBUG [hconnection-0x585d597-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:55:17,352 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47712, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:55:17,354 DEBUG [hconnection-0x232bb1a2-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:55:17,354 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47742, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:55:17,354 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47704, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:55:17,354 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47734, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:55:17,356 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47724, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:55:17,356 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47782, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:55:17,358 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47756, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:55:17,358 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:47768, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:55:17,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:17,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:17,359 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:17,359 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:17,359 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:17,361 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:17,363 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,363 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:17,365 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,365 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:17,365 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,365 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:17,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:17,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:17,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:17,369 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:17,372 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8be8843d5bbff3d4194de143513dc64b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-08 02:55:17,373 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=A 2023-04-08 02:55:17,373 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:17,373 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=B 2023-04-08 02:55:17,373 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:17,373 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=C 2023-04-08 02:55:17,373 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:17,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:17,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:17,374 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:17,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:17,383 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:17,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922577382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922577382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,384 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:17,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922577383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1680922517377, max timestamp=1680922517382Num uniques 859; 2023-04-08 02:55:17,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:17,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-08 02:55:17,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47734 deadline: 1680922577384, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1680922517377, max timestamp=1680922517382Num uniques 859; 2023-04-08 02:55:17,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-08 02:55:17,384 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=459, cellSize=21015, totalHeapSize=47976, min timestamp=1680922517377, max timestamp=1680922517382Num uniques 859; 2023-04-08 02:55:17,385 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=21015, suffix heap size=117064,new segment heap size=47976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=459 2023-04-08 02:55:17,385 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:17,385 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:17,385 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:17,385 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:17,385 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:17,387 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:17,387 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:17,387 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:17,387 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:17,387 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:17,387 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:17,387 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:17,387 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:17,387 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,387 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:17,387 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:17,389 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:17,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:17,389 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:17,391 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,391 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:17,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:17,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:17,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:17,393 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4734e727[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:17,393 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:17,393 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,395 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,396 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:17,396 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:17,396 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:17,396 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,396 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:17,397 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:17,397 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,397 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:17,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:17,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:17,397 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:17,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:17,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:17,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:17,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:17,399 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:17,399 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:17,400 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:17,400 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:17,400 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:17,400 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:17,401 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,401 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:17,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:17,401 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:17,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:17,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:17,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:17,402 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:17,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,403 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:17,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,404 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:17,404 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47700 deadline: 1680922577404, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,405 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:17,406 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:17,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:17,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:17,406 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:17,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:17,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:17,406 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,406 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:17,406 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:17,406 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,406 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:17,406 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:17,406 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,406 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:17,406 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:17,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:17,406 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,411 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408d246150270d144b9a1fcfbe45f36ca36 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408d246150270d144b9a1fcfbe45f36ca36 2023-04-08 02:55:17,432 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/5e242e9852be4046a9669294a12af6ce 2023-04-08 02:55:17,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:17,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922577485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:17,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922577485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,487 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:17,487 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47734 deadline: 1680922577485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,488 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:17,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922577485, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,506 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:17,507 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:17,508 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:17,509 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:17,509 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47700 deadline: 1680922577505, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:17,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:17,513 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:17,513 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:17,513 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:17,513 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:17,514 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,514 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:17,521 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:17,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:17,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:17,521 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:17,522 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,522 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:17,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:17,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:17,522 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:17,526 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:17,526 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:17,526 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:17,526 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:17,526 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:17,530 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:17,530 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:17,530 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:17,530 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:17,531 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:17,531 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:17,531 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:17,531 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:17,531 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,532 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:17,532 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:17,533 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,533 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:17,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:17,534 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:17,534 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:17,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:17,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:17,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:17,535 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f040692[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:17,535 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:17,535 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,538 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,538 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:17,538 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:17,538 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:17,538 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,538 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:17,539 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:17,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:17,539 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:17,539 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:17,539 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:17,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:17,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:17,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,540 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:17,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:17,541 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:17,541 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:17,542 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:17,542 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:17,542 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:17,542 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:17,545 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,545 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:17,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:17,545 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:17,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:17,545 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:17,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:17,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:17,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,547 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,547 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:17,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,549 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,549 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:17,549 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:17,549 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:17,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:17,550 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:17,549 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:17,550 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:17,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:17,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:17,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:17,550 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,550 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,550 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,550 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,550 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:17,550 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:17,550 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:17,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:17,650 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:17,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:17,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:17,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:17,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:17,654 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:17,654 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:17,654 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:17,654 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:17,654 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,654 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:17,655 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:17,655 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,655 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:17,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:17,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:17,656 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,656 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:17,656 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:17,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:17,657 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:17,658 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:17,658 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:17,658 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:17,658 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:17,658 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:17,658 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:17,658 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:17,658 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:17,658 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:17,658 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:17,658 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:17,659 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:17,659 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:17,659 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,659 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:17,659 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:17,660 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,660 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:17,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:17,660 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,660 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:17,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:17,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:17,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:17,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:17,662 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17f5333f[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:17,662 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:17,662 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,664 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,665 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:17,665 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:17,665 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:17,665 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,665 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:17,666 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:17,666 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:17,666 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,666 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:17,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:17,666 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:17,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:17,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:17,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:17,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,669 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:17,670 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:17,670 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:17,670 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:17,670 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:17,670 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:17,670 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:17,672 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,672 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,672 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:17,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:17,672 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:17,672 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:17,673 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:17,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:17,673 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,673 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:17,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,674 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:17,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,680 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,680 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:17,680 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:17,680 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,680 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,680 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:17,680 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:17,680 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:17,680 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,680 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:17,680 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:17,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:17,680 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:17,680 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:17,681 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:17,681 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:17,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:17,690 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:17,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922577688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,690 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:17,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922577688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,690 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:17,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922577689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,692 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:17,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47734 deadline: 1680922577689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:17,712 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47700 deadline: 1680922577711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,781 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:17,781 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:17,783 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:17,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:17,784 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:17,784 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:17,785 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:17,785 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:17,785 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:17,785 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,785 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:17,786 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,786 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:17,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:17,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:17,786 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,786 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:17,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:17,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:17,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:17,787 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:17,787 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:17,788 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:17,788 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:17,788 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:17,788 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:17,788 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:17,788 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:17,788 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:17,788 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:17,788 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:17,788 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:17,788 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:17,788 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,788 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:17,788 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:17,790 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,790 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:17,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:17,790 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,790 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:17,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:17,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:17,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:17,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:17,792 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:17,792 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1ca25bb5[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:17,792 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,793 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,793 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,793 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:17,793 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:17,793 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:17,794 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,794 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:17,795 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,795 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:17,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:17,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:17,795 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:17,795 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:17,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:17,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:17,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:17,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:17,798 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:17,798 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:17,798 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:17,798 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:17,798 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:17,798 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:17,799 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,799 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,799 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,799 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:17,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:17,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:17,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,800 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:17,800 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:17,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:17,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:17,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,801 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:17,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,802 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,803 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:17,803 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,803 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:17,803 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:17,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:17,804 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:17,803 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:17,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:17,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:17,803 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:17,804 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:17,804 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:17,804 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:17,804 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,804 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,804 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:17,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:17,804 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,857 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/ea8bf85a01ec447d8d63ed885c143793 2023-04-08 02:55:17,904 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:17,905 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:17,906 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:17,907 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:17,908 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:17,908 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:17,908 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:17,908 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:17,908 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:17,908 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,908 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:17,909 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:17,909 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:17,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:17,910 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,910 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:17,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:17,910 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:17,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:17,911 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:17,911 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:17,911 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:17,911 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:17,911 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:17,911 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:17,911 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:17,911 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:17,911 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:17,911 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:17,911 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:17,911 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:17,911 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:17,912 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:17,912 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,912 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:17,912 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:17,914 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,914 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:17,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:17,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:17,914 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,914 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:17,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:17,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:17,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:17,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ccf9ad9[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:17,916 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:17,916 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,917 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,917 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:17,917 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:17,917 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:17,917 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,917 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:17,918 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:17,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:17,919 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:17,919 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:17,919 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:17,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:17,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:17,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:17,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:17,921 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:17,921 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:17,921 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:17,921 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:17,921 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:17,921 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:17,923 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,923 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:17,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:17,923 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:17,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:17,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:17,923 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:17,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:17,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,925 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:17,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:17,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,927 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,927 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:17,927 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:17,927 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,927 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,928 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:17,928 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:17,928 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:17,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:17,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:17,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:17,928 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:17,928 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:17,928 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:17,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:17,928 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:17,928 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:17,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:17,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:17,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:17,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922577991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:17,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922577991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:17,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922577991, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:17,995 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:17,995 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 13 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47734 deadline: 1680922577995, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,014 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:18,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47700 deadline: 1680922578013, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,028 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:18,029 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:18,032 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:18,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:18,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:18,034 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:18,034 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:18,034 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:18,034 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:18,035 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:18,035 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:18,036 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,037 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,037 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:18,037 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:18,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:18,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:18,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:18,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:18,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:18,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:18,040 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:18,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:18,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:18,041 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:18,041 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:18,041 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:18,041 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:18,041 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:18,041 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:18,041 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:18,041 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:18,041 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:18,041 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:18,041 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:18,041 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,041 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:18,042 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:18,043 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,043 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:18,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:18,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:18,043 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:18,043 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:18,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:18,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:18,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:18,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:18,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:18,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3802344[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:18,045 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:18,045 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:18,047 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:18,047 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:18,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:18,047 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:18,047 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:18,047 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:18,047 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,047 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:18,049 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:18,049 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:18,049 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,049 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:18,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:18,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:18,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:18,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:18,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:18,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:18,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:18,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:18,052 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:18,052 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:18,052 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:18,052 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:18,052 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:18,052 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:18,055 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:18,055 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:18,055 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:18,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:18,055 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:18,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:18,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:18,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:18,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:18,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:18,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:18,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:18,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:18,057 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:18,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:18,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:18,058 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:18,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,064 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,064 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:18,064 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:18,064 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:18,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:18,064 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:18,064 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:18,064 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:18,064 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:18,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:18,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:18,064 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:18,065 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:18,064 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:18,065 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:18,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:18,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:18,165 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:18,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:18,167 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:18,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:18,168 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:18,168 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:18,168 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:18,168 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:18,168 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:18,168 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:18,168 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:18,173 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:18,173 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,173 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:18,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:18,174 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,174 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:18,174 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:18,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:18,174 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:18,175 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:18,175 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:18,175 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:18,175 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:18,175 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:18,175 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:18,175 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:18,175 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:18,175 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:18,175 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:18,175 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:18,175 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:18,175 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:18,175 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,175 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:18,175 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:18,177 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:18,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:18,177 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:18,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:18,177 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:18,177 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:18,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:18,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:18,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:18,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:18,178 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d5294c0[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:18,178 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:18,178 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:18,179 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:18,179 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:18,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:18,180 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:18,180 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:18,180 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,180 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:18,180 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:18,182 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,182 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:18,182 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:18,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:18,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:18,182 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:18,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:18,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:18,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:18,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:18,184 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:18,184 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:18,184 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:18,184 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:18,184 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:18,186 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:18,186 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:18,186 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:18,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:18,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:18,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:18,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:18,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:18,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:18,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:18,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:18,187 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:18,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:18,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:18,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:18,188 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:18,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:18,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,190 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,190 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:18,190 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:18,190 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:18,190 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:18,190 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:18,190 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:18,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:18,190 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:18,190 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:18,190 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:18,191 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:18,190 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:18,191 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:18,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:18,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:18,191 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:18,191 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:18,270 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/5dff424c7dce4cb488e8f94159bad612 2023-04-08 02:55:18,274 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/5e242e9852be4046a9669294a12af6ce as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/5e242e9852be4046a9669294a12af6ce 2023-04-08 02:55:18,277 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/5e242e9852be4046a9669294a12af6ce, entries=150, sequenceid=16, filesize=25.1 K 2023-04-08 02:55:18,278 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/ea8bf85a01ec447d8d63ed885c143793 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/ea8bf85a01ec447d8d63ed885c143793 2023-04-08 02:55:18,281 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/ea8bf85a01ec447d8d63ed885c143793, entries=150, sequenceid=16, filesize=11.6 K 2023-04-08 02:55:18,282 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/5dff424c7dce4cb488e8f94159bad612 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/5dff424c7dce4cb488e8f94159bad612 2023-04-08 02:55:18,285 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/5dff424c7dce4cb488e8f94159bad612, entries=150, sequenceid=16, filesize=11.6 K 2023-04-08 02:55:18,286 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=127.47 KB/130530 for 8be8843d5bbff3d4194de143513dc64b in 914ms, sequenceid=16, compaction requested=false 2023-04-08 02:55:18,286 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:18,291 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:18,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:18,293 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:18,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:18,294 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:18,294 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:18,294 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:18,294 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:18,294 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:18,295 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:18,295 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:18,296 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:18,296 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:18,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:18,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:18,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:18,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:18,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:18,297 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:18,297 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:18,297 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:18,297 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:18,297 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:18,297 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:18,298 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:18,298 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:18,298 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:18,298 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:18,298 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,298 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:18,298 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8be8843d5bbff3d4194de143513dc64b 3/3 column families, dataSize=127.47 KB heapSize=336.94 KB 2023-04-08 02:55:18,298 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=A 2023-04-08 02:55:18,298 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:18,298 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=B 2023-04-08 02:55:18,298 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:18,298 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=C 2023-04-08 02:55:18,298 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:18,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:18,500 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:18,514 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:18,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922578510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,515 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:18,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922578510, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,520 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:18,520 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47734 deadline: 1680922578514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:18,521 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922578515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,523 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:18,523 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47700 deadline: 1680922578517, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,616 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:18,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922578616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,619 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:18,619 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922578616, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:18,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47734 deadline: 1680922578621, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,625 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:18,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922578622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,714 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408fc0e371d74784157ace9f2ba8d499dd9 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408fc0e371d74784157ace9f2ba8d499dd9 2023-04-08 02:55:18,715 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-08 02:55:18,730 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=46, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/3f4c3bd1dc52463ca794dca0559887da 2023-04-08 02:55:18,767 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/79e1becd6b73458ca1c6a2d0d88448c6 2023-04-08 02:55:18,822 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:18,822 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922578818, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,824 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:18,825 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922578821, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,838 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:18,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47734 deadline: 1680922578832, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:18,839 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:18,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922578834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:19,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922579123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,129 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:19,129 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922579128, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,145 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:19,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922579142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,145 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:19,146 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47734 deadline: 1680922579142, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,193 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=46 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/9661dac69b7c4096a04d8c4b3f0e3862 2023-04-08 02:55:19,197 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/3f4c3bd1dc52463ca794dca0559887da as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/3f4c3bd1dc52463ca794dca0559887da 2023-04-08 02:55:19,201 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/3f4c3bd1dc52463ca794dca0559887da, entries=150, sequenceid=46, filesize=25.1 K 2023-04-08 02:55:19,202 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/79e1becd6b73458ca1c6a2d0d88448c6 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/79e1becd6b73458ca1c6a2d0d88448c6 2023-04-08 02:55:19,206 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/79e1becd6b73458ca1c6a2d0d88448c6, entries=150, sequenceid=46, filesize=11.6 K 2023-04-08 02:55:19,207 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/9661dac69b7c4096a04d8c4b3f0e3862 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/9661dac69b7c4096a04d8c4b3f0e3862 2023-04-08 02:55:19,211 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/9661dac69b7c4096a04d8c4b3f0e3862, entries=150, sequenceid=46, filesize=11.6 K 2023-04-08 02:55:19,213 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~336.89 KB/344976, currentSize=60.38 KB/61830 for 8be8843d5bbff3d4194de143513dc64b in 915ms, sequenceid=46, compaction requested=false 2023-04-08 02:55:19,213 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:19,213 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:19,213 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:19,213 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:19,213 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,213 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:19,213 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:19,215 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:19,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:19,215 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:19,215 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:19,215 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:19,215 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:19,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:19,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:19,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:19,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:19,217 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33a31f2c[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:19,217 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:19,217 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:19,218 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:19,218 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:19,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:19,218 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:19,218 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:19,218 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,219 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:19,218 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:19,220 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:19,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:19,220 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:19,220 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:19,220 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:19,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:19,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:19,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:19,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:19,223 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:19,223 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:19,223 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:19,223 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:19,223 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:19,223 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:19,226 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:19,226 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:19,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:19,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:19,226 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:19,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:19,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:19,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:19,226 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:19,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:19,226 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:19,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:19,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:19,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:19,228 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:19,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:19,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:19,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,231 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:19,231 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:19,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:19,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:19,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:19,231 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:19,232 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:19,232 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:19,232 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:19,232 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:19,232 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:19,232 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,232 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:19,232 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:19,232 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:19,232 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:19,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:19,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:19,332 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:19,333 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:19,334 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:19,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:19,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:19,335 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:19,335 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:19,335 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:19,335 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:19,336 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:19,336 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:19,337 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:19,337 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:19,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:19,337 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,338 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:19,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:19,338 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:19,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:19,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:19,338 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:19,338 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:19,339 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:19,339 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:19,339 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:19,339 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:19,339 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:19,339 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:19,339 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:19,339 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:19,339 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8be8843d5bbff3d4194de143513dc64b 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-08 02:55:19,339 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=A 2023-04-08 02:55:19,339 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:19,339 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=B 2023-04-08 02:55:19,339 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:19,339 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=C 2023-04-08 02:55:19,339 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:19,356 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304085839627db72845f78c1517e08a5b06a8 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304085839627db72845f78c1517e08a5b06a8 2023-04-08 02:55:19,364 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=58, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/a9de419bf17444eb90fd56a4ef07968f 2023-04-08 02:55:19,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:19,530 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:19,603 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:19,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47700 deadline: 1680922579598, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,630 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:19,630 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922579627, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,637 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:19,637 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922579632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,651 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:19,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47734 deadline: 1680922579646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:19,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922579648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,711 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:19,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47700 deadline: 1680922579704, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:19,774 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/626491655b19477e87b536fcad936495 2023-04-08 02:55:19,915 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:19,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47700 deadline: 1680922579913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,185 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=58 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/6710cd7c3a4c438890a62cde699f457e 2023-04-08 02:55:20,190 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/a9de419bf17444eb90fd56a4ef07968f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/a9de419bf17444eb90fd56a4ef07968f 2023-04-08 02:55:20,195 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/a9de419bf17444eb90fd56a4ef07968f, entries=150, sequenceid=58, filesize=25.1 K 2023-04-08 02:55:20,201 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/626491655b19477e87b536fcad936495 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/626491655b19477e87b536fcad936495 2023-04-08 02:55:20,204 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/626491655b19477e87b536fcad936495, entries=150, sequenceid=58, filesize=11.6 K 2023-04-08 02:55:20,205 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/6710cd7c3a4c438890a62cde699f457e as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/6710cd7c3a4c438890a62cde699f457e 2023-04-08 02:55:20,208 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/6710cd7c3a4c438890a62cde699f457e, entries=150, sequenceid=58, filesize=11.6 K 2023-04-08 02:55:20,208 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 8be8843d5bbff3d4194de143513dc64b in 869ms, sequenceid=58, compaction requested=true 2023-04-08 02:55:20,208 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:20,208 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:20,209 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:20,209 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:20,209 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,209 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:20,209 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:20,210 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,210 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:20,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:20,211 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:20,211 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:20,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:20,211 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:20,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:20,212 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@333a42d3[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:20,212 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:20,212 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,213 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,213 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:20,213 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:20,213 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:20,215 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,215 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:20,217 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:20,217 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:20,217 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,217 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:20,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:20,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:20,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:20,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:20,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:20,221 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:20,221 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:20,221 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:20,221 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:20,221 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:20,221 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:20,222 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,223 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,223 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:20,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:20,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:20,224 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:20,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:20,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,224 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:20,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:20,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922519587, max timestamp=1680922520222Num uniques 859; 2023-04-08 02:55:20,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:55:20,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922519587, max timestamp=1680922520222Num uniques 859; 2023-04-08 02:55:20,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:55:20,226 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8be8843d5bbff3d4194de143513dc64b 3/3 column families, dataSize=58.14 KB heapSize=164.17 KB 2023-04-08 02:55:20,226 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=A 2023-04-08 02:55:20,226 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:20,226 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=B 2023-04-08 02:55:20,226 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:20,226 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,226 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=C 2023-04-08 02:55:20,227 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:20,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:20,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,237 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,239 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:20,239 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:20,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:20,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:20,240 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:20,239 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:20,240 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:20,240 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:20,240 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:20,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:20,240 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:20,240 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:20,240 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:20,240 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,240 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,240 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,240 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,275 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408ce0507f8586a459bbcfbe2ef48f3a779 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408ce0507f8586a459bbcfbe2ef48f3a779 2023-04-08 02:55:20,291 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=79, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/c136c9f5586646338ceae3a588cbcf74 2023-04-08 02:55:20,340 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:20,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:20,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:20,344 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922520314, max timestamp=1680922520342Num uniques 859; 2023-04-08 02:55:20,344 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:55:20,344 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922520314, max timestamp=1680922520342Num uniques 859; 2023-04-08 02:55:20,344 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:55:20,344 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922520314, max timestamp=1680922520342Num uniques 859; 2023-04-08 02:55:20,344 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:55:20,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:20,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:20,345 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:20,345 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:20,345 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:20,345 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:20,346 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,346 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:20,347 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,347 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:20,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:20,347 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,347 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:20,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:20,348 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:20,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:20,348 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:20,348 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:20,348 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:20,349 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:20,349 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:20,349 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:20,352 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:20,352 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:20,352 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:20,352 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:20,352 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:20,352 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:20,352 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:20,352 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:20,352 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,353 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:20,353 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:20,358 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,358 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:20,358 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,358 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:20,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:20,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:20,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:20,359 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:20,359 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3122ddf3[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:20,360 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,360 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,361 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,361 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:20,361 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:20,361 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,361 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:20,361 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:20,362 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:20,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:20,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:20,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,363 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:20,363 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:20,363 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:20,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:20,364 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:20,364 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:20,364 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:20,364 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:20,364 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:20,364 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:20,367 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,367 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:20,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:20,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,367 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:20,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:20,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:20,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:20,367 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:20,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,369 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:20,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,372 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,372 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:20,372 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:20,372 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,372 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,372 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,372 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:20,372 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:20,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:20,373 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:20,373 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:20,373 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:20,373 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:20,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:20,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:20,373 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:20,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:20,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:20,442 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:20,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47700 deadline: 1680922580436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,473 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:20,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:20,475 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:20,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:20,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:20,476 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:20,476 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:20,476 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:20,476 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:20,476 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,477 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:20,478 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,478 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:20,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:20,478 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,478 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:20,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:20,478 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:20,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:20,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:20,479 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:20,479 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:20,479 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:20,479 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:20,479 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:20,479 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:20,479 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:20,479 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:20,479 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:20,479 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:20,479 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:20,480 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:20,480 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:20,480 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,480 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:20,480 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:20,481 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,481 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:20,481 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,481 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:20,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:20,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:20,483 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@187fab83[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:20,483 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:20,483 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,485 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,485 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:20,485 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,485 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:20,485 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:20,485 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:20,486 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,486 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:20,487 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:20,487 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:20,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:20,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:20,489 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:20,489 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:20,489 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:20,489 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:20,489 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:20,489 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:20,490 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,490 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,490 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:20,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:20,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:20,491 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:20,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:20,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:20,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:20,492 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:20,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,494 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,494 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:20,494 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:20,494 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:20,494 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:20,494 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:20,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:20,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:20,495 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:20,494 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,495 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,495 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,494 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:20,495 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:20,495 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:20,495 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:20,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:20,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:20,545 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:20,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47700 deadline: 1680922580544, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,595 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:20,596 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:20,597 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:20,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:20,598 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:20,598 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:20,598 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:20,598 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:20,598 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:20,599 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,599 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:20,601 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,601 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:20,601 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:20,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:20,602 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,602 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:20,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:20,602 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:20,602 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:20,602 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:20,603 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:20,603 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:20,603 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:20,603 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:20,603 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:20,603 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:20,603 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:20,603 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:20,603 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:20,603 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:20,603 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:20,603 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:20,603 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,603 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:20,603 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:20,605 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,605 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:20,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:20,605 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,605 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:20,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:20,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:20,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,606 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:20,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:20,606 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@338e8ddf[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:20,606 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:20,606 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,607 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,607 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,607 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,608 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:20,608 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:20,608 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,608 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:20,608 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:20,609 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:20,609 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:20,609 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,609 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:20,609 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:20,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:20,611 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:20,611 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:20,611 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:20,611 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:20,612 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:20,612 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:20,618 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,618 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,619 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:20,619 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,619 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:20,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:20,620 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:20,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,622 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:20,622 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,622 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:20,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:20,622 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:20,622 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:20,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:20,622 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:20,622 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:20,623 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:20,623 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:20,623 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,623 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,623 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:20,623 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:20,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:20,623 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,623 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,623 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:20,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:20,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922580637, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,650 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:20,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922580647, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:20,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922580660, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,665 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:20,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47734 deadline: 1680922580661, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,707 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/2159c0f3754d4e1585022fd127265188 2023-04-08 02:55:20,718 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=79 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/9361748db1ed465a85392ac661f3c718 2023-04-08 02:55:20,723 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/c136c9f5586646338ceae3a588cbcf74 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/c136c9f5586646338ceae3a588cbcf74 2023-04-08 02:55:20,723 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:20,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:20,725 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:20,730 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/c136c9f5586646338ceae3a588cbcf74, entries=200, sequenceid=79, filesize=31.9 K 2023-04-08 02:55:20,731 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/2159c0f3754d4e1585022fd127265188 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/2159c0f3754d4e1585022fd127265188 2023-04-08 02:55:20,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:20,732 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:20,732 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:20,732 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:20,732 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:20,732 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:20,733 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,733 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:20,734 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:20,734 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,734 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:20,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:20,735 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,735 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:20,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:20,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:20,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:20,736 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:20,748 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:20,748 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:20,748 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:20,748 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:20,748 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:20,749 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:20,749 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:20,749 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:20,749 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:20,749 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:20,749 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:20,749 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:20,749 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:20,749 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,749 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:20,749 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:20,751 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,751 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:20,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:20,751 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,751 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:20,751 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:20,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47700 deadline: 1680922580747, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:20,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:20,756 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52436da7[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:20,756 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:20,756 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,756 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/2159c0f3754d4e1585022fd127265188, entries=150, sequenceid=79, filesize=11.6 K 2023-04-08 02:55:20,757 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/9361748db1ed465a85392ac661f3c718 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/9361748db1ed465a85392ac661f3c718 2023-04-08 02:55:20,758 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,759 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,759 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:20,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:20,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:20,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:20,763 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,763 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:20,763 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:20,763 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:20,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:20,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:20,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:20,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:20,766 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:20,767 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:20,767 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:20,767 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:20,767 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:20,768 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/9361748db1ed465a85392ac661f3c718, entries=150, sequenceid=79, filesize=11.6 K 2023-04-08 02:55:20,768 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~58.14 KB/59540, heapSize ~164.13 KB/168064, currentSize=127.47 KB/130530 for 8be8843d5bbff3d4194de143513dc64b in 542ms, sequenceid=79, compaction requested=true 2023-04-08 02:55:20,769 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:20,769 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,769 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:20,769 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:20,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,769 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:55:20,769 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:55:20,769 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:20,769 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:20,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:20,769 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:20,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:20,770 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:20,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,770 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:55:20,770 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 8be8843d5bbff3d4194de143513dc64b/B is initiating minor compaction (all files) 2023-04-08 02:55:20,770 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8be8843d5bbff3d4194de143513dc64b/B in TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:20,770 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/ea8bf85a01ec447d8d63ed885c143793, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/79e1becd6b73458ca1c6a2d0d88448c6, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/626491655b19477e87b536fcad936495, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/2159c0f3754d4e1585022fd127265188] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp, totalSize=46.4 K 2023-04-08 02:55:20,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,771 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting ea8bf85a01ec447d8d63ed885c143793, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680922517370 2023-04-08 02:55:20,771 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 109924 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:55:20,771 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 8be8843d5bbff3d4194de143513dc64b/A is initiating minor compaction (all files) 2023-04-08 02:55:20,771 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8be8843d5bbff3d4194de143513dc64b/A in TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:20,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:20,771 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/5e242e9852be4046a9669294a12af6ce, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/3f4c3bd1dc52463ca794dca0559887da, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/a9de419bf17444eb90fd56a4ef07968f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/c136c9f5586646338ceae3a588cbcf74] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp, totalSize=107.3 K 2023-04-08 02:55:20,771 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,771 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 79e1becd6b73458ca1c6a2d0d88448c6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1680922517380 2023-04-08 02:55:20,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,772 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 5e242e9852be4046a9669294a12af6ce, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680922517370 2023-04-08 02:55:20,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,772 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 3f4c3bd1dc52463ca794dca0559887da, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1680922517380 2023-04-08 02:55:20,772 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 626491655b19477e87b536fcad936495, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1680922518504 2023-04-08 02:55:20,772 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting a9de419bf17444eb90fd56a4ef07968f, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1680922518504 2023-04-08 02:55:20,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:20,772 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 2159c0f3754d4e1585022fd127265188, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1680922519587 2023-04-08 02:55:20,773 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting c136c9f5586646338ceae3a588cbcf74, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1680922519587 2023-04-08 02:55:20,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:20,774 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,774 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:20,774 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:20,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:20,774 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:20,774 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:20,774 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:20,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:20,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:20,774 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:20,775 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:20,775 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:20,775 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,775 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:20,775 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,775 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:20,775 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:20,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:20,782 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 8be8843d5bbff3d4194de143513dc64b#B#compaction#368 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:20,800 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8be8843d5bbff3d4194de143513dc64b#A#compaction#369 average throughput is 1.31 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:20,875 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:20,875 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:20,877 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:20,878 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:20,879 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:20,879 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:20,879 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:20,879 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:20,879 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:20,879 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,879 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:20,881 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,881 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:20,881 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:20,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:20,882 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:20,882 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:20,882 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:20,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:20,882 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:20,883 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:20,883 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:20,883 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:20,883 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:20,883 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:20,885 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:20,885 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:20,885 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:20,885 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:20,885 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8be8843d5bbff3d4194de143513dc64b 3/3 column families, dataSize=127.47 KB heapSize=365.06 KB 2023-04-08 02:55:20,885 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=A 2023-04-08 02:55:20,885 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:20,885 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=B 2023-04-08 02:55:20,885 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:20,885 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=C 2023-04-08 02:55:20,885 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:20,902 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408214d79f4e1f64ae79b227c679999d7c4 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408214d79f4e1f64ae79b227c679999d7c4 2023-04-08 02:55:21,059 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:21,059 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:21,095 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:21,095 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47700 deadline: 1680922581095, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,202 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:21,202 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47700 deadline: 1680922581196, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,207 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/1331db03ee74421fad9569f2785d58f9 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/1331db03ee74421fad9569f2785d58f9 2023-04-08 02:55:21,214 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8be8843d5bbff3d4194de143513dc64b/B of 8be8843d5bbff3d4194de143513dc64b into 1331db03ee74421fad9569f2785d58f9(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:21,214 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:21,215 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., storeName=8be8843d5bbff3d4194de143513dc64b/B, priority=12, startTime=1680922520769; duration=0sec 2023-04-08 02:55:21,215 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:21,215 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:55:21,216 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:55:21,216 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 8be8843d5bbff3d4194de143513dc64b/C is initiating minor compaction (all files) 2023-04-08 02:55:21,217 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8be8843d5bbff3d4194de143513dc64b/C in TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:21,217 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/5dff424c7dce4cb488e8f94159bad612, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/9661dac69b7c4096a04d8c4b3f0e3862, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/6710cd7c3a4c438890a62cde699f457e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/9361748db1ed465a85392ac661f3c718] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp, totalSize=46.4 K 2023-04-08 02:55:21,217 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 5dff424c7dce4cb488e8f94159bad612, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680922517370 2023-04-08 02:55:21,217 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 9661dac69b7c4096a04d8c4b3f0e3862, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=46, earliestPutTs=1680922517380 2023-04-08 02:55:21,218 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 6710cd7c3a4c438890a62cde699f457e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=58, earliestPutTs=1680922518504 2023-04-08 02:55:21,218 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 9361748db1ed465a85392ac661f3c718, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1680922519587 2023-04-08 02:55:21,229 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 8be8843d5bbff3d4194de143513dc64b#C#compaction#371 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:21,230 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/153ce75049044085b601c881407e2c2f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/153ce75049044085b601c881407e2c2f 2023-04-08 02:55:21,235 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8be8843d5bbff3d4194de143513dc64b/A of 8be8843d5bbff3d4194de143513dc64b into 153ce75049044085b601c881407e2c2f(size=25.3 K), total size for store is 25.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:21,235 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:21,235 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., storeName=8be8843d5bbff3d4194de143513dc64b/A, priority=12, startTime=1680922520769; duration=0sec 2023-04-08 02:55:21,236 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:21,242 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/edffb163eca7493981b9d94db7a53e91 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/edffb163eca7493981b9d94db7a53e91 2023-04-08 02:55:21,247 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8be8843d5bbff3d4194de143513dc64b/C of 8be8843d5bbff3d4194de143513dc64b into edffb163eca7493981b9d94db7a53e91(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:21,247 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:21,247 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., storeName=8be8843d5bbff3d4194de143513dc64b/C, priority=12, startTime=1680922520769; duration=0sec 2023-04-08 02:55:21,247 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:21,307 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=115, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/998785f6ddce4510abb8deab24526ccb 2023-04-08 02:55:21,317 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/48a5f51d0d734396a43b78330a1dabaa 2023-04-08 02:55:21,331 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=115 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/c1b61b2253274e889990846209926776 2023-04-08 02:55:21,335 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/998785f6ddce4510abb8deab24526ccb as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/998785f6ddce4510abb8deab24526ccb 2023-04-08 02:55:21,338 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/998785f6ddce4510abb8deab24526ccb, entries=150, sequenceid=115, filesize=25.1 K 2023-04-08 02:55:21,339 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/48a5f51d0d734396a43b78330a1dabaa as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/48a5f51d0d734396a43b78330a1dabaa 2023-04-08 02:55:21,342 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/48a5f51d0d734396a43b78330a1dabaa, entries=150, sequenceid=115, filesize=11.6 K 2023-04-08 02:55:21,343 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/c1b61b2253274e889990846209926776 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/c1b61b2253274e889990846209926776 2023-04-08 02:55:21,349 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/c1b61b2253274e889990846209926776, entries=150, sequenceid=115, filesize=11.6 K 2023-04-08 02:55:21,350 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~365.02 KB/373776, currentSize=53.67 KB/54960 for 8be8843d5bbff3d4194de143513dc64b in 465ms, sequenceid=115, compaction requested=false 2023-04-08 02:55:21,350 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:21,350 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:21,350 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:21,350 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:21,350 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,353 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:21,353 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:21,355 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,355 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:21,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:21,355 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:21,355 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:21,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:21,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7529ef59[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:21,361 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:21,361 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,362 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,362 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:21,362 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:21,362 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:21,364 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:21,364 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:21,364 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:21,366 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,367 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:21,367 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:21,367 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:21,367 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:21,367 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:21,367 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:21,367 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:21,371 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,371 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:21,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,371 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:21,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:21,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,373 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:21,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:21,375 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:21,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,379 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,379 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:21,379 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:21,379 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,379 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,379 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:21,379 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:21,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:21,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:21,379 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:21,379 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,380 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:21,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:21,380 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:21,380 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:21,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:21,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:21,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:21,407 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:21,408 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8be8843d5bbff3d4194de143513dc64b 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-08 02:55:21,408 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=A 2023-04-08 02:55:21,408 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:21,408 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=B 2023-04-08 02:55:21,408 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:21,409 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=C 2023-04-08 02:55:21,409 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:21,428 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040869a300c1c1114b9cb141a7f834d2978e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040869a300c1c1114b9cb141a7f834d2978e 2023-04-08 02:55:21,436 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=130, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/f47e35890d97493293e04b9bafef8722 2023-04-08 02:55:21,445 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/811462e7d2d348c8b5444880dfe53b8f 2023-04-08 02:55:21,460 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=130 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/32e48b7e1c0c4f2ab1eb0d8471083c18 2023-04-08 02:55:21,464 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/f47e35890d97493293e04b9bafef8722 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/f47e35890d97493293e04b9bafef8722 2023-04-08 02:55:21,468 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/f47e35890d97493293e04b9bafef8722, entries=200, sequenceid=130, filesize=32.0 K 2023-04-08 02:55:21,468 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/811462e7d2d348c8b5444880dfe53b8f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/811462e7d2d348c8b5444880dfe53b8f 2023-04-08 02:55:21,471 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/811462e7d2d348c8b5444880dfe53b8f, entries=150, sequenceid=130, filesize=11.7 K 2023-04-08 02:55:21,472 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/32e48b7e1c0c4f2ab1eb0d8471083c18 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/32e48b7e1c0c4f2ab1eb0d8471083c18 2023-04-08 02:55:21,475 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/32e48b7e1c0c4f2ab1eb0d8471083c18, entries=150, sequenceid=130, filesize=11.7 K 2023-04-08 02:55:21,476 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=87.22 KB/89310 for 8be8843d5bbff3d4194de143513dc64b in 69ms, sequenceid=130, compaction requested=true 2023-04-08 02:55:21,476 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:21,476 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:21,476 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:55:21,476 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:55:21,476 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:21,476 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:21,477 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 84366 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:55:21,477 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 8be8843d5bbff3d4194de143513dc64b/A is initiating minor compaction (all files) 2023-04-08 02:55:21,477 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35846 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:55:21,477 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8be8843d5bbff3d4194de143513dc64b/A in TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:21,477 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 8be8843d5bbff3d4194de143513dc64b/B is initiating minor compaction (all files) 2023-04-08 02:55:21,477 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/153ce75049044085b601c881407e2c2f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/998785f6ddce4510abb8deab24526ccb, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/f47e35890d97493293e04b9bafef8722] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp, totalSize=82.4 K 2023-04-08 02:55:21,477 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8be8843d5bbff3d4194de143513dc64b/B in TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:21,477 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/1331db03ee74421fad9569f2785d58f9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/48a5f51d0d734396a43b78330a1dabaa, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/811462e7d2d348c8b5444880dfe53b8f] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp, totalSize=35.0 K 2023-04-08 02:55:21,477 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 153ce75049044085b601c881407e2c2f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1680922519587 2023-04-08 02:55:21,478 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 1331db03ee74421fad9569f2785d58f9, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1680922519587 2023-04-08 02:55:21,478 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 998785f6ddce4510abb8deab24526ccb, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1680922520418 2023-04-08 02:55:21,478 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 48a5f51d0d734396a43b78330a1dabaa, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1680922520418 2023-04-08 02:55:21,478 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting f47e35890d97493293e04b9bafef8722, keycount=200, bloomtype=ROW, size=32.0 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680922521065 2023-04-08 02:55:21,478 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 811462e7d2d348c8b5444880dfe53b8f, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680922521065 2023-04-08 02:55:21,480 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:21,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:21,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:21,482 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8be8843d5bbff3d4194de143513dc64b 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-08 02:55:21,482 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=A 2023-04-08 02:55:21,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:21,482 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:21,483 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=B 2023-04-08 02:55:21,483 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:21,483 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=C 2023-04-08 02:55:21,483 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:21,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:21,484 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:21,484 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:21,484 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:21,484 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:21,484 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:21,484 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,484 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:21,486 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:21,486 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,486 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:21,486 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,486 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:21,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:21,487 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:21,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:21,487 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:21,487 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:21,487 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:21,487 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:21,487 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:21,487 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:21,488 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:21,488 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:21,488 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:21,488 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:21,488 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:21,488 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:21,488 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:21,488 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:21,488 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,488 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:21,488 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:21,489 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 8be8843d5bbff3d4194de143513dc64b#B#compaction#377 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:21,490 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,490 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:21,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:21,490 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,491 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:21,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:21,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:21,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@10ef61cc[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:21,492 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:21,493 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,495 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,495 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,496 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:21,496 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:21,496 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:21,496 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,496 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:21,497 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:21,497 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:21,497 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,497 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:21,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:21,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:21,499 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:21,500 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:21,500 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:21,500 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:21,500 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:21,501 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,501 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,501 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:21,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:21,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:21,501 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8be8843d5bbff3d4194de143513dc64b#A#compaction#378 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:21,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:21,501 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:21,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,503 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,507 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:21,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:21,507 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:21,507 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:21,507 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:21,507 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:21,507 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:21,507 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:21,507 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,507 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:21,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:21,507 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:21,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:21,507 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,507 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,507 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,507 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:21,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:21,510 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408fecabb72cfb844cabc7503fc080ab620 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408fecabb72cfb844cabc7503fc080ab620 2023-04-08 02:55:21,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:21,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47700 deadline: 1680922581542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,607 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:21,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:21,609 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:21,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:21,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:21,610 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:21,610 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:21,610 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:21,610 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:21,611 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,611 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:21,613 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:21,613 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:21,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:21,614 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,614 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:21,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:21,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:21,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:21,615 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:21,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:21,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:21,616 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:21,616 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:21,616 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:21,616 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:21,616 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:21,616 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:21,616 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:21,616 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:21,616 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:21,616 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:21,616 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:21,616 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,616 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:21,616 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:21,618 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,618 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:21,618 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:21,618 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:21,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:21,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:21,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:21,619 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@57c8f67b[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:21,619 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:21,619 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,620 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,621 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:21,621 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:21,621 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:21,621 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,621 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:21,622 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:21,622 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:21,622 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,622 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:21,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:21,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:21,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:21,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,624 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:21,624 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:21,624 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:21,624 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:21,624 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:21,624 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:21,624 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:21,628 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,628 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,628 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:21,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:21,628 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:21,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:21,630 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:21,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:21,630 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,631 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:21,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,635 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,635 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:21,635 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:21,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:21,636 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:21,635 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:21,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:21,636 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:21,635 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:21,636 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,636 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,636 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:21,636 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:21,636 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,636 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:21,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:21,652 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:21,653 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47700 deadline: 1680922581650, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,736 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:21,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:21,738 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:21,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:21,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:21,739 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:21,739 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:21,740 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:21,740 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:21,740 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,740 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:21,741 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,741 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:21,741 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:21,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:21,741 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,741 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:21,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:21,742 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:21,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:21,742 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:21,742 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:21,742 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:21,743 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:21,743 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:21,743 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:21,743 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:21,743 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:21,743 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:21,743 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:21,743 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:21,744 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:21,744 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:21,744 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:21,744 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,744 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:21,744 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:21,746 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,746 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:21,747 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,747 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:21,747 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:21,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:21,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,749 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:21,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:21,750 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1eaf9770[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:21,750 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:21,750 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,751 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,751 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,751 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:21,751 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:21,751 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:21,751 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,751 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:21,754 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:21,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:21,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:21,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:21,755 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,755 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:21,755 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:21,755 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:21,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:21,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:21,757 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:21,757 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:21,757 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:21,757 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:21,757 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:21,757 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:21,758 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:21,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:21,759 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,759 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:21,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:21,759 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:21,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:21,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:21,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,761 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:21,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,764 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,764 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:21,764 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:21,764 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,764 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,764 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:21,764 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:21,764 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,765 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:21,764 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:21,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:21,765 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:21,765 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:21,765 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:21,765 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:21,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:21,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:21,857 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:21,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47700 deadline: 1680922581855, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,865 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:21,866 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:21,867 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:21,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:21,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:21,868 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:21,868 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:21,868 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:21,868 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:21,869 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,869 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:21,870 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,870 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:21,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:21,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:21,870 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,870 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:21,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:21,871 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:21,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:21,871 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:21,871 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:21,871 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:21,872 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:21,872 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:21,872 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:21,872 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:21,872 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:21,872 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:21,872 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:21,872 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:21,872 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:21,872 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:21,872 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:21,872 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,872 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:21,872 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:21,874 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,874 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:21,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:21,874 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,874 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:21,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:21,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:21,874 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:21,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:21,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2446f0db[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:21,875 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:21,875 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,876 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,876 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,876 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:21,876 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,876 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:21,876 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:21,877 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:21,879 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,879 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:21,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:21,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:21,879 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:21,880 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:21,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:21,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:21,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:21,882 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:21,882 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:21,882 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:21,882 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:21,882 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:21,882 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:21,883 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,883 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,883 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:21,883 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,884 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:21,884 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:21,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,885 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:21,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,887 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,887 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:21,887 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:21,887 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:21,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:21,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:21,887 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:21,887 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:21,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:21,887 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:21,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:21,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:21,887 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:21,887 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,888 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:21,888 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:21,888 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:21,888 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:21,908 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/7c073e081ccf417182b58773a647a7b0 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/7c073e081ccf417182b58773a647a7b0 2023-04-08 02:55:21,912 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8be8843d5bbff3d4194de143513dc64b/B of 8be8843d5bbff3d4194de143513dc64b into 7c073e081ccf417182b58773a647a7b0(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:21,912 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:21,912 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., storeName=8be8843d5bbff3d4194de143513dc64b/B, priority=13, startTime=1680922521476; duration=0sec 2023-04-08 02:55:21,913 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:21,913 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:55:21,913 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35846 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:55:21,914 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 8be8843d5bbff3d4194de143513dc64b/C is initiating minor compaction (all files) 2023-04-08 02:55:21,914 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8be8843d5bbff3d4194de143513dc64b/C in TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:21,914 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/edffb163eca7493981b9d94db7a53e91, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/c1b61b2253274e889990846209926776, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/32e48b7e1c0c4f2ab1eb0d8471083c18] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp, totalSize=35.0 K 2023-04-08 02:55:21,914 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting edffb163eca7493981b9d94db7a53e91, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=79, earliestPutTs=1680922519587 2023-04-08 02:55:21,915 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting c1b61b2253274e889990846209926776, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=115, earliestPutTs=1680922520418 2023-04-08 02:55:21,915 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 32e48b7e1c0c4f2ab1eb0d8471083c18, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680922521065 2023-04-08 02:55:21,920 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=147, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/29fcdd96a0c148959e87a373fb7599df 2023-04-08 02:55:21,923 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 8be8843d5bbff3d4194de143513dc64b#C#compaction#380 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:21,927 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/6564c1724fc14cd8be95892afc2e6121 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/6564c1724fc14cd8be95892afc2e6121 2023-04-08 02:55:21,931 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8be8843d5bbff3d4194de143513dc64b/A of 8be8843d5bbff3d4194de143513dc64b into 6564c1724fc14cd8be95892afc2e6121(size=25.4 K), total size for store is 25.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:21,931 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:21,931 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., storeName=8be8843d5bbff3d4194de143513dc64b/A, priority=13, startTime=1680922521476; duration=0sec 2023-04-08 02:55:21,932 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:21,988 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:21,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:21,990 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:21,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:21,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:21,999 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:22,000 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:22,000 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:22,000 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:22,000 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,000 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,001 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:22,001 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:22,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:22,002 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,002 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:22,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,002 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:22,003 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:22,003 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:22,003 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:22,003 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:22,003 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:22,004 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:22,004 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:22,004 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:22,004 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:22,004 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:22,004 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:22,004 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:22,004 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:22,004 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,004 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:22,004 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:22,006 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:22,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:22,006 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:22,007 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,007 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:22,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:22,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:22,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:22,008 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4fcc149e[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:22,008 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:22,008 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,009 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,009 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,009 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:22,009 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:22,009 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:22,010 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,010 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:22,011 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,011 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:22,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:22,011 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:22,011 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:22,014 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:22,014 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:22,014 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:22,014 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:22,014 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:22,014 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:22,015 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,015 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:22,015 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,015 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:22,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:22,016 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:22,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,017 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:22,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,020 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:22,020 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:22,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:22,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:22,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:22,021 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,021 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,021 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,021 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,021 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:22,021 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:22,021 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:22,021 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:22,021 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,021 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:22,021 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,021 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:22,121 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:22,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:22,123 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:22,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:22,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:22,124 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:22,124 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:22,124 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:22,124 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:22,124 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,124 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,125 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:22,125 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:22,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:22,126 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,126 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:22,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,126 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:22,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,127 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:22,127 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:22,127 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:22,127 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:22,127 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:22,127 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:22,127 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:22,127 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:22,127 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:22,127 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:22,127 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:22,127 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:22,127 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:22,127 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:22,127 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,127 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:22,127 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:22,129 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,129 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:22,129 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,129 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:22,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:22,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:22,130 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f274832[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:22,130 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:22,131 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,131 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,131 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,132 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:22,132 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,132 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:22,132 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:22,132 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:22,134 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,134 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:22,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:22,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:22,134 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:22,134 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:22,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:22,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:22,136 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:22,136 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:22,136 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:22,136 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:22,136 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:22,136 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:22,138 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,138 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,138 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:22,138 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:22,138 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:22,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:22,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,139 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:22,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,141 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:22,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:22,141 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:22,142 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:22,142 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:22,142 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,142 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:22,142 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:22,142 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:22,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:22,142 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:22,142 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,142 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,142 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,142 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,142 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:22,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:22,162 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:22,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47700 deadline: 1680922582159, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,242 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:22,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:22,244 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:22,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:22,245 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:22,245 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:22,245 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:22,245 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:22,245 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:22,246 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,246 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,247 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,247 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:22,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:22,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:22,247 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,247 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:22,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,248 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:22,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,248 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:22,248 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:22,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:22,248 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:22,248 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:22,248 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:22,249 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:22,249 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:22,249 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:22,249 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:22,249 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:22,249 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:22,249 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:22,249 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:22,249 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,249 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:22,249 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:22,252 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,252 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:22,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:22,252 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,252 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:22,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:22,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:22,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:22,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:22,255 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3dd150b1[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:22,255 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:22,255 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,256 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,257 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:22,257 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,257 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:22,257 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:22,257 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:22,258 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:22,258 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:22,258 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,258 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:22,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:22,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:22,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:22,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:22,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:22,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:22,261 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:22,261 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:22,261 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:22,261 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:22,261 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:22,261 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:22,262 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,262 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,262 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:22,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:22,262 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:22,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:22,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:22,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,264 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:22,265 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:22,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,268 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,268 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,268 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,268 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,268 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:22,268 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:22,268 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:22,268 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:22,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:22,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:22,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:22,268 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,268 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,268 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:22,268 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:22,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:22,269 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:22,269 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:22,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:22,330 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/13eed4c94b294bcea7b242e6541cc153 2023-04-08 02:55:22,334 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/d1c075e5002c441a99d6cb840a85bda2 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/d1c075e5002c441a99d6cb840a85bda2 2023-04-08 02:55:22,339 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8be8843d5bbff3d4194de143513dc64b/C of 8be8843d5bbff3d4194de143513dc64b into d1c075e5002c441a99d6cb840a85bda2(size=11.9 K), total size for store is 11.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:22,339 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:22,339 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., storeName=8be8843d5bbff3d4194de143513dc64b/C, priority=13, startTime=1680922521476; duration=0sec 2023-04-08 02:55:22,339 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:22,369 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:22,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:22,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:22,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:22,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:22,372 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:22,372 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:22,372 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:22,372 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:22,372 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,372 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,373 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,373 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:22,373 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:22,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:22,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,374 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,374 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:22,374 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:22,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:22,375 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:22,375 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:22,375 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:22,375 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:22,375 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:22,376 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:22,376 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:22,376 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:22,376 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:22,376 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:22,376 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:22,376 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:22,377 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:22,377 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,377 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:22,377 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:22,378 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:22,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,379 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:22,379 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,380 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:22,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:22,380 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2e02bff8[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:22,380 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:22,380 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,381 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,381 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,381 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:22,381 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:22,381 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:22,381 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,381 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:22,383 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,383 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:22,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:22,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:22,383 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:22,383 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:22,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:22,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:22,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:22,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:22,386 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:22,386 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:22,386 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:22,386 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:22,386 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:22,386 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:22,387 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,387 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,387 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:22,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:22,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:22,387 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,388 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:22,388 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:22,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:22,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:22,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,389 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:22,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,391 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,391 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,391 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,391 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,391 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:22,391 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:22,391 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:22,391 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:22,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:22,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:22,391 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,391 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:22,391 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:22,391 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:22,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:22,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:22,391 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:22,492 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:22,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:22,494 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:22,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:22,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:22,494 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:22,495 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:22,495 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:22,495 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:22,495 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,495 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,500 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,500 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:22,500 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:22,500 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,500 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:22,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,501 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:22,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,501 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:22,501 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:22,501 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:22,501 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:22,501 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:22,501 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:22,502 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:22,502 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:22,502 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:22,502 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:22,502 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:22,502 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:22,502 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:22,502 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:22,502 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,502 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:22,502 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:22,503 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,503 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:22,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:22,504 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,504 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:22,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:22,505 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:22,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1453642[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:22,505 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,506 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,506 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:22,506 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:22,506 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:22,506 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,506 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:22,507 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,507 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:22,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:22,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:22,507 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:22,508 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:22,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:22,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:22,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:22,510 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:22,510 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:22,510 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:22,510 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:22,510 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:22,510 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:22,512 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,512 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,512 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:22,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:22,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:22,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:22,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:22,513 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:22,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,514 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:22,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,517 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:22,517 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:22,517 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:22,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:22,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:22,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:22,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:22,517 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,517 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,517 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,517 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,517 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,518 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,518 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,518 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:22,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:22,523 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:22,523 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:22,523 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:22,623 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:22,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:22,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:22,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:22,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:22,627 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:22,627 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:22,627 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:22,627 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:22,627 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,628 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,629 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:22,629 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:22,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:22,629 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,629 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:22,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:22,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,630 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:22,630 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:22,630 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:22,630 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:22,630 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:22,630 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:22,630 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:22,630 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:22,631 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:22,631 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:22,631 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:22,631 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:22,631 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:22,631 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:22,631 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,631 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:22,631 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:22,632 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,632 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:22,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:22,633 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,633 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:22,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:22,634 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64c68880[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:22,634 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:22,634 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,636 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,636 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:22,636 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:22,636 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:22,636 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,636 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:22,637 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:22,637 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,637 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:22,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:22,637 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:22,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:22,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:22,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:22,640 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:22,640 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:22,640 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:22,640 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:22,640 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:22,640 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:22,642 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,642 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,642 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:22,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,642 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:22,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:22,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:22,643 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:22,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:22,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,645 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,645 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:22,645 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,646 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:22,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:22,646 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:22,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:22,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:22,652 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,652 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,652 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:22,652 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:22,652 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:22,652 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:22,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:22,652 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:22,652 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:22,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:22,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922582654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,658 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:22,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922582654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,661 DEBUG [Thread-2577] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4148 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:55:22,662 DEBUG [Thread-2575] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4152 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:55:22,668 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:22,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47700 deadline: 1680922582666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,671 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:22,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47734 deadline: 1680922582667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:22,673 DEBUG [Thread-2579] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4158 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:55:22,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922582669, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,674 DEBUG [Thread-2571] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4159 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:55:22,741 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=147 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/48f755102cc54276b4e4ddfee5905b78 2023-04-08 02:55:22,745 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/29fcdd96a0c148959e87a373fb7599df as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/29fcdd96a0c148959e87a373fb7599df 2023-04-08 02:55:22,748 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/29fcdd96a0c148959e87a373fb7599df, entries=200, sequenceid=147, filesize=32.1 K 2023-04-08 02:55:22,749 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/13eed4c94b294bcea7b242e6541cc153 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/13eed4c94b294bcea7b242e6541cc153 2023-04-08 02:55:22,752 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/13eed4c94b294bcea7b242e6541cc153, entries=150, sequenceid=147, filesize=11.8 K 2023-04-08 02:55:22,752 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:22,753 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/48f755102cc54276b4e4ddfee5905b78 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/48f755102cc54276b4e4ddfee5905b78 2023-04-08 02:55:22,753 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:22,755 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:22,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:22,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:22,756 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:22,756 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:22,756 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:22,756 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:22,757 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,757 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,759 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:22,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:22,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:22,759 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/48f755102cc54276b4e4ddfee5905b78, entries=150, sequenceid=147, filesize=11.8 K 2023-04-08 02:55:22,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,759 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:22,760 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 8be8843d5bbff3d4194de143513dc64b in 1278ms, sequenceid=147, compaction requested=false 2023-04-08 02:55:22,760 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:22,760 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:22,760 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:22,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:22,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:22,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:22,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:22,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:22,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:22,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:22,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:22,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:22,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:22,761 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:22,761 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:22,761 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8be8843d5bbff3d4194de143513dc64b 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-08 02:55:22,761 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=A 2023-04-08 02:55:22,761 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:22,761 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=B 2023-04-08 02:55:22,761 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:22,762 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=C 2023-04-08 02:55:22,762 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:22,777 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304083296659b77b046148e5e85b591515139 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304083296659b77b046148e5e85b591515139 2023-04-08 02:55:22,786 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=165, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/90f24717e0e5434b8d8a65a8a5c5928f 2023-04-08 02:55:22,834 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/9f04d1f1f6824ae0ae0fcaaadb7a5b9f 2023-04-08 02:55:23,259 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=165 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/d39e8f81335341d984d2ee464f734285 2023-04-08 02:55:23,264 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/90f24717e0e5434b8d8a65a8a5c5928f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/90f24717e0e5434b8d8a65a8a5c5928f 2023-04-08 02:55:23,268 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/90f24717e0e5434b8d8a65a8a5c5928f, entries=150, sequenceid=165, filesize=25.3 K 2023-04-08 02:55:23,268 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/9f04d1f1f6824ae0ae0fcaaadb7a5b9f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/9f04d1f1f6824ae0ae0fcaaadb7a5b9f 2023-04-08 02:55:23,272 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/9f04d1f1f6824ae0ae0fcaaadb7a5b9f, entries=150, sequenceid=165, filesize=11.8 K 2023-04-08 02:55:23,272 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/d39e8f81335341d984d2ee464f734285 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/d39e8f81335341d984d2ee464f734285 2023-04-08 02:55:23,275 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/d39e8f81335341d984d2ee464f734285, entries=150, sequenceid=165, filesize=11.8 K 2023-04-08 02:55:23,276 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=0 B/0 for 8be8843d5bbff3d4194de143513dc64b in 515ms, sequenceid=165, compaction requested=true 2023-04-08 02:55:23,276 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:23,276 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:23,276 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:23,276 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:23,276 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,277 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:23,277 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:23,278 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,278 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:23,278 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:23,278 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,278 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:23,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:23,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:23,280 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@53251871[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:23,280 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:23,280 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,281 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,281 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,281 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,281 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:23,281 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:23,281 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:23,281 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,282 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:23,283 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,283 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:23,283 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:23,283 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:23,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:23,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:23,288 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:23,288 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:23,288 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:23,288 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:23,288 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:23,288 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:23,291 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:23,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:23,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:23,292 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:23,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,293 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:23,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:23,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,296 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,296 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:23,296 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:23,296 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:23,296 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:23,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:23,296 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,296 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:23,296 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,296 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:23,296 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:23,296 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:23,296 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,296 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,297 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,297 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:23,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:23,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:23,297 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:23,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:23,396 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:23,397 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:23,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:23,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:23,403 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:23,403 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:23,403 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:23,403 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:23,403 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:23,403 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,404 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:23,405 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,405 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:23,405 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:23,405 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,405 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:23,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:23,406 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:23,406 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:23,406 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:23,406 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:23,407 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:23,407 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:23,407 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:23,407 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:23,407 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:23,407 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:23,407 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:23,407 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:23,407 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:23,407 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:23,407 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:23,407 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,408 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:23,408 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:23,412 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,412 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:23,412 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:23,413 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,413 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:23,413 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:23,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:23,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,414 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:23,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:23,414 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68e39092[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:23,414 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:23,415 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,416 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,416 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:23,416 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,416 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:23,416 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:23,416 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:23,418 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,418 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:23,418 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:23,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:23,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:23,418 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:23,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:23,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:23,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:23,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:23,420 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:23,420 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:23,421 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:23,421 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:23,421 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:23,421 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:23,422 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,422 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:23,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,422 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:23,423 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:23,423 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:23,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:23,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:23,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:23,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:23,424 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:23,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:23,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,427 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:23,427 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,427 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:23,427 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:23,427 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:23,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:23,428 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:23,427 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:23,428 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:23,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:23,428 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,428 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,428 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,428 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,428 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:23,428 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:23,428 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:23,428 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:23,428 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:23,528 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:23,529 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:23,532 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:23,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:23,540 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:23,540 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:23,540 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:23,540 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:23,540 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:23,543 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,543 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:23,544 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,544 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:23,545 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:23,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:23,545 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,545 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:23,545 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:23,545 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:23,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:23,546 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:23,546 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:23,546 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:23,546 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:23,546 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:23,547 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:23,548 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:23,548 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:23,548 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:23,548 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:23,548 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:23,548 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:23,548 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:23,548 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:23,548 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,548 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:23,548 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:23,550 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,550 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:23,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:23,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:23,551 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,551 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:23,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:23,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:23,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:23,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:23,561 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@403c4cea[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:23,561 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:23,561 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,562 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,562 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,563 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:23,563 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:23,563 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:23,563 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,563 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:23,566 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:23,566 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:23,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:23,566 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:23,566 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:23,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:23,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:23,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:23,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:23,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:23,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:23,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:23,569 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:23,569 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:23,569 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:23,569 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:23,569 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:23,571 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,571 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,571 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:23,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:23,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:23,572 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:23,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:23,572 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:23,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:23,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:23,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:23,575 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:23,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:23,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,578 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,578 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:23,578 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:23,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:23,578 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:23,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:23,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:23,578 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:23,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:23,578 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:23,578 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:23,578 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:23,578 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,578 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,578 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,578 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:23,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:23,580 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:23,678 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:23,679 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:23,684 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:23,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:23,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:23,685 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:23,686 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:23,686 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:23,686 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:23,686 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,686 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:23,688 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,688 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:23,688 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:23,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:23,688 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,688 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:23,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:23,689 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:23,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:23,690 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:23,690 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:23,690 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:23,690 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:23,691 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:23,691 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:23,696 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:23,696 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:23,696 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:23,696 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:23,696 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:23,696 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:23,696 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:23,696 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:23,696 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,696 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:23,696 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:23,698 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,698 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:23,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:23,699 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,699 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:23,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:23,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:23,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:23,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:23,702 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@106e4ee5[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:23,702 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:23,702 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,703 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,703 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:23,704 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:23,704 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:23,704 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,704 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:23,706 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:23,706 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:23,706 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,706 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:23,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:23,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:23,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:23,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:23,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:23,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:23,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:23,711 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:23,711 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:23,711 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:23,711 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:23,711 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:23,711 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:23,714 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:23,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:23,715 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:23,716 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,716 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,717 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:23,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:23,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:23,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,719 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:23,719 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:23,719 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,719 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:23,719 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:23,720 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,720 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,720 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:23,720 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:23,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:23,720 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:23,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:23,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:23,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:23,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:23,720 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:23,720 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:23,820 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:23,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:23,823 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:23,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:23,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:23,824 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:23,824 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:23,824 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:23,824 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:23,825 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,825 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:23,826 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:23,826 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:23,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:23,826 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:23,827 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:23,827 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:23,827 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:23,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:23,827 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:23,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:23,828 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:23,828 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:23,828 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:23,828 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:23,828 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:23,832 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:23,832 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:23,832 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:23,832 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:23,833 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8be8843d5bbff3d4194de143513dc64b 3/3 column families, dataSize=13.42 KB heapSize=40.59 KB 2023-04-08 02:55:23,833 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=A 2023-04-08 02:55:23,833 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:23,833 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=B 2023-04-08 02:55:23,833 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:23,834 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=C 2023-04-08 02:55:23,834 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:23,872 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408a9e3b9a2e76641678c5ff09c4c4e6efb to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408a9e3b9a2e76641678c5ff09c4c4e6efb 2023-04-08 02:55:23,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:23,889 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:23,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922523953, max timestamp=1680922523965Num uniques 859; 2023-04-08 02:55:23,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:55:23,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922523953, max timestamp=1680922523965Num uniques 859; 2023-04-08 02:55:23,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922523953, max timestamp=1680922523965Num uniques 859; 2023-04-08 02:55:23,966 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:55:23,967 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:55:24,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680922523999, max timestamp=1680922524061Num uniques 1018; 2023-04-08 02:55:24,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680922523999, max timestamp=1680922524061Num uniques 1018; 2023-04-08 02:55:24,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680922523999, max timestamp=1680922524061Num uniques 1018; 2023-04-08 02:55:24,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-08 02:55:24,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-08 02:55:24,064 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-08 02:55:24,151 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680922524111, max timestamp=1680922524149Num uniques 1027; 2023-04-08 02:55:24,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680922524111, max timestamp=1680922524149Num uniques 1027; 2023-04-08 02:55:24,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-08 02:55:24,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680922524111, max timestamp=1680922524149Num uniques 1027; 2023-04-08 02:55:24,154 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-08 02:55:24,152 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-08 02:55:24,267 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680922524241, max timestamp=1680922524266Num uniques 1036; 2023-04-08 02:55:24,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-08 02:55:24,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680922524241, max timestamp=1680922524266Num uniques 1036; 2023-04-08 02:55:24,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-08 02:55:24,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680922524241, max timestamp=1680922524266Num uniques 1036; 2023-04-08 02:55:24,268 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-08 02:55:24,278 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=173, memsize=4.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/a4d1a40f2b1a4860bba3ceef568b05e3 2023-04-08 02:55:24,287 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=173 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/34eb14d15d0341b99b8c2ad0ab743cf1 2023-04-08 02:55:24,348 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680922524321, max timestamp=1680922524346Num uniques 1045; 2023-04-08 02:55:24,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-08 02:55:24,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680922524321, max timestamp=1680922524346Num uniques 1045; 2023-04-08 02:55:24,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-08 02:55:24,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680922524321, max timestamp=1680922524346Num uniques 1045; 2023-04-08 02:55:24,349 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-08 02:55:24,459 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680922524425, max timestamp=1680922524457Num uniques 1054; 2023-04-08 02:55:24,460 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-08 02:55:24,460 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680922524425, max timestamp=1680922524457Num uniques 1054; 2023-04-08 02:55:24,460 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-08 02:55:24,460 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=154, cellSize=7050, totalHeapSize=16256, min timestamp=1680922524425, max timestamp=1680922524457Num uniques 1054; 2023-04-08 02:55:24,460 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=7050, suffix heap size=137584,new segment heap size=16256 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=154 2023-04-08 02:55:24,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680922524506, max timestamp=1680922524557Num uniques 1013; 2023-04-08 02:55:24,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-08 02:55:24,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680922524506, max timestamp=1680922524557Num uniques 1013; 2023-04-08 02:55:24,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-08 02:55:24,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680922524506, max timestamp=1680922524557Num uniques 1013; 2023-04-08 02:55:24,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46388, new segment data size=7458, suffix heap size=133320,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1013, new segment cells count=163 2023-04-08 02:55:24,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680922524584, max timestamp=1680922524654Num uniques 1022; 2023-04-08 02:55:24,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680922524584, max timestamp=1680922524654Num uniques 1022; 2023-04-08 02:55:24,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-08 02:55:24,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680922524584, max timestamp=1680922524654Num uniques 1022; 2023-04-08 02:55:24,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-08 02:55:24,657 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-08 02:55:24,696 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=4.47 KB at sequenceid=173 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/916edf6e8df34fd09b14d9400b7c61cd 2023-04-08 02:55:24,699 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/a4d1a40f2b1a4860bba3ceef568b05e3 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/a4d1a40f2b1a4860bba3ceef568b05e3 2023-04-08 02:55:24,702 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/a4d1a40f2b1a4860bba3ceef568b05e3, entries=100, sequenceid=173, filesize=18.5 K 2023-04-08 02:55:24,703 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/34eb14d15d0341b99b8c2ad0ab743cf1 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/34eb14d15d0341b99b8c2ad0ab743cf1 2023-04-08 02:55:24,706 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/34eb14d15d0341b99b8c2ad0ab743cf1, entries=100, sequenceid=173, filesize=9.4 K 2023-04-08 02:55:24,706 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/916edf6e8df34fd09b14d9400b7c61cd as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/916edf6e8df34fd09b14d9400b7c61cd 2023-04-08 02:55:24,709 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/916edf6e8df34fd09b14d9400b7c61cd, entries=100, sequenceid=173, filesize=9.4 K 2023-04-08 02:55:24,710 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~13.42 KB/13740, heapSize ~40.55 KB/41520, currentSize=100.63 KB/103050 for 8be8843d5bbff3d4194de143513dc64b in 877ms, sequenceid=173, compaction requested=true 2023-04-08 02:55:24,710 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:24,710 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:24,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:24,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:24,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:24,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:24,713 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:24,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:24,713 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:24,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:24,713 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:24,713 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:24,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:24,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:24,714 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:24,714 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8be8843d5bbff3d4194de143513dc64b 3/3 column families, dataSize=107.34 KB heapSize=304.08 KB 2023-04-08 02:55:24,714 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=A 2023-04-08 02:55:24,714 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:24,714 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=B 2023-04-08 02:55:24,714 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:24,714 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=C 2023-04-08 02:55:24,714 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:24,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:24,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:24,715 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6bf32a09[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:24,715 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:24,715 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:24,716 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:24,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:24,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:24,716 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:24,716 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:24,716 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:24,717 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,717 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:24,721 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,721 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:24,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:24,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:24,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:24,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:24,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:24,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:24,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:24,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:24,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:24,724 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:24,724 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:24,724 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:24,724 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:24,724 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:24,726 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408a2739c9528454ede98a08ae3f0e166b6 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408a2739c9528454ede98a08ae3f0e166b6 2023-04-08 02:55:24,727 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:24,727 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:24,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:24,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:24,727 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:24,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:24,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:24,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:24,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:24,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:24,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:24,727 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:24,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:24,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,733 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:24,734 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:24,734 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=326, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/9c867c1d12b5449bb81be16cbc6d75e0 2023-04-08 02:55:24,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,738 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,738 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:24,738 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:24,738 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,738 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:24,738 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:24,738 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:24,739 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:24,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:24,739 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:24,739 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:24,739 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:24,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:24,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:24,739 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:24,739 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:24,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:24,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:24,790 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:24,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 303 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47700 deadline: 1680922584783, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,839 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:24,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:24,841 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:24,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:24,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:24,842 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:24,842 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:24,842 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:24,842 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:24,843 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:24,843 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:24,844 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:24,844 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,844 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:24,844 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,844 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:24,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:24,845 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:24,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:24,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:24,845 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:24,845 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:24,845 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:24,845 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:24,845 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:24,845 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:24,846 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:24,846 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:24,846 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:24,846 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:24,846 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:24,846 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:24,846 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:24,846 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:24,846 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,846 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:24,846 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:24,847 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:24,847 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:24,848 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:24,848 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:24,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:24,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:24,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:24,849 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fe050e5[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:24,849 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:24,849 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:24,852 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:24,852 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:24,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:24,852 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:24,852 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:24,852 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,852 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:24,852 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:24,853 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,853 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:24,853 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:24,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:24,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:24,853 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:24,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:24,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:24,855 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:24,855 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:24,856 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:24,856 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:24,856 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:24,856 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:24,857 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:24,857 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:24,857 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:24,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:24,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:24,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:24,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:24,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:24,857 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:24,857 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:24,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:24,859 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:24,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,861 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,861 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:24,861 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:24,861 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:24,861 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:24,861 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:24,861 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:24,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:24,861 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:24,861 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:24,861 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:24,861 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:24,861 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:24,861 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:24,861 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:24,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:24,896 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:24,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 305 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47700 deadline: 1680922584891, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,961 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:24,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:24,965 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:24,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:24,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:24,966 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:24,966 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:24,966 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:24,966 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:24,967 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:24,967 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:24,969 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:24,969 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,969 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:24,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:24,969 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:24,969 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:24,969 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:24,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:24,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:24,970 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:24,970 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:24,970 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:24,970 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:24,970 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:24,970 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:24,970 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:24,970 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:24,970 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:24,971 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:24,971 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:24,971 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:24,971 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:24,971 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:24,971 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,971 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:24,971 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:24,973 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:24,973 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:24,973 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:24,974 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:24,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:24,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:24,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:24,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:24,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:24,985 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3d1904ef[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:24,985 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:24,985 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:24,987 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:24,987 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:24,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:24,987 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:24,987 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,987 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:24,989 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,989 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:24,989 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:24,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:24,989 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:24,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:24,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:24,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:24,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:24,993 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:24,993 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:24,993 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:24,993 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:24,993 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:24,996 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:24,996 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:24,996 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:24,996 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:24,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:24,997 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:24,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:24,998 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:24,999 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:25,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,003 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,003 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:25,003 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:25,003 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,004 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,004 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,004 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:25,004 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:25,004 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:25,004 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:25,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:25,004 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:25,004 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:25,004 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:25,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:25,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:25,102 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:25,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 307 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47700 deadline: 1680922585098, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,104 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:25,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:25,106 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:25,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:25,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:25,108 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:25,108 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:25,108 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:25,108 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:25,108 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,108 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:25,110 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,110 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:25,110 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:25,110 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,110 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:25,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:25,111 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:25,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:25,111 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:25,111 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:25,111 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:25,111 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:25,112 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:25,112 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:25,112 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:25,112 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:25,112 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:25,112 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:25,112 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:25,112 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:25,112 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:25,112 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:25,112 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,112 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:25,112 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:25,114 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,114 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:25,114 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,114 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:25,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:25,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:25,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:25,116 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5fe27ccf[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:25,116 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:25,116 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,118 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,118 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,118 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:25,118 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:25,118 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:25,118 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,118 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:25,119 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:25,119 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,119 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:25,120 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:25,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:25,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:25,122 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:25,122 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:25,122 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:25,122 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:25,122 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:25,122 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:25,126 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,126 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:25,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,127 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:25,127 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:25,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:25,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:25,129 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:25,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:25,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,133 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,133 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:25,133 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:25,133 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,133 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,133 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:25,133 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:25,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:25,134 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:25,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:25,134 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:25,133 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,134 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,134 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:25,134 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:25,134 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:25,134 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:25,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:25,147 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/665807a0694a4cd49c52de79bfb59972 2023-04-08 02:55:25,234 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:25,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:25,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:25,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:25,237 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:25,237 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:25,237 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:25,237 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:25,237 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:25,238 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,238 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:25,239 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,239 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:25,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:25,239 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,239 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:25,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:25,240 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:25,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:25,240 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:25,240 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:25,240 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:25,240 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:25,240 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:25,240 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:25,241 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:25,241 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:25,241 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:25,241 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:25,241 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:25,241 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:25,241 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:25,241 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:25,241 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,241 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:25,241 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:25,242 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,242 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:25,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:25,243 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,243 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:25,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:25,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:25,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:25,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:25,244 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d33a658[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:25,244 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:25,244 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,246 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,246 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:25,246 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:25,246 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,246 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:25,246 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:25,248 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:25,248 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:25,248 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:25,248 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:25,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:25,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:25,250 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:25,250 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:25,250 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:25,250 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:25,250 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:25,251 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:25,252 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,252 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:25,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:25,252 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:25,252 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:25,253 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:25,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,254 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:25,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:25,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,255 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:25,256 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:25,256 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:25,256 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:25,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:25,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:25,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:25,256 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:25,256 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:25,256 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,256 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,256 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,256 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,256 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:25,256 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:25,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:25,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:25,356 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:25,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:25,358 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:25,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:25,360 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:25,360 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:25,360 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:25,360 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:25,360 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:25,360 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,360 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:25,361 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,361 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:25,362 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:25,362 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,362 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:25,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:25,362 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:25,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:25,363 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:25,363 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:25,363 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:25,363 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:25,363 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:25,363 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:25,363 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:25,363 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:25,363 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:25,363 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:25,363 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:25,363 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:25,364 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:25,366 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:25,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:25,367 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:25,367 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,367 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:25,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:25,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:25,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30c9ac63[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:25,368 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:25,368 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,369 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,369 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:25,369 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,369 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:25,369 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:25,369 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:25,371 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:25,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:25,371 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:25,371 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:25,371 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:25,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:25,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:25,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:25,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:25,373 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:25,373 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:25,373 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:25,373 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:25,373 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:25,373 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:25,375 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,375 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:25,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:25,375 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:25,375 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:25,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:25,376 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:25,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:25,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,379 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:25,380 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:25,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:25,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:25,380 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:25,380 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,380 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:25,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:25,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:25,380 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:25,380 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:25,380 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,380 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,380 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:25,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:25,380 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,380 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:25,380 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:25,380 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,407 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:25,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 309 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47700 deadline: 1680922585403, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,480 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:25,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:25,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:25,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:25,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:25,483 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:25,483 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:25,483 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:25,483 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:25,484 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,484 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:25,485 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:25,485 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:25,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:25,486 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:25,486 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,486 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:25,486 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:25,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:25,486 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:25,487 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:25,487 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:25,487 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:25,487 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:25,487 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:25,487 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:25,487 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:25,489 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,489 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:25,489 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,489 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:25,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:25,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:25,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:25,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:25,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:25,491 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4502e8e9[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:25,491 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:25,491 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,492 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,492 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:25,492 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:25,492 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,492 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:25,492 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:25,496 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,496 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:25,496 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:25,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:25,496 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:25,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:25,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:25,499 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:25,499 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:25,499 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:25,499 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:25,499 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:25,499 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:25,500 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,500 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,501 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,501 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:25,500 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:25,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,502 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:25,509 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,509 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:25,509 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:25,509 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:25,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:25,509 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:25,509 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:25,509 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:25,509 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:25,509 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:25,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:25,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:25,509 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:25,509 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:25,509 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:25,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:25,564 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=326 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/276ce84520c1494396f2c6b363e07e26 2023-04-08 02:55:25,569 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/9c867c1d12b5449bb81be16cbc6d75e0 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/9c867c1d12b5449bb81be16cbc6d75e0 2023-04-08 02:55:25,572 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/9c867c1d12b5449bb81be16cbc6d75e0, entries=200, sequenceid=326, filesize=32.3 K 2023-04-08 02:55:25,573 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/665807a0694a4cd49c52de79bfb59972 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/665807a0694a4cd49c52de79bfb59972 2023-04-08 02:55:25,576 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/665807a0694a4cd49c52de79bfb59972, entries=150, sequenceid=326, filesize=11.9 K 2023-04-08 02:55:25,577 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/276ce84520c1494396f2c6b363e07e26 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/276ce84520c1494396f2c6b363e07e26 2023-04-08 02:55:25,581 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/276ce84520c1494396f2c6b363e07e26, entries=150, sequenceid=326, filesize=11.9 K 2023-04-08 02:55:25,581 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~304.03 KB/311328, currentSize=73.80 KB/75570 for 8be8843d5bbff3d4194de143513dc64b in 867ms, sequenceid=326, compaction requested=true 2023-04-08 02:55:25,581 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:25,582 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:25,582 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 02:55:25,582 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 02:55:25,582 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:25,582 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:25,583 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 136802 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 02:55:25,583 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58076 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 02:55:25,583 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 8be8843d5bbff3d4194de143513dc64b/A is initiating minor compaction (all files) 2023-04-08 02:55:25,583 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 8be8843d5bbff3d4194de143513dc64b/B is initiating minor compaction (all files) 2023-04-08 02:55:25,583 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8be8843d5bbff3d4194de143513dc64b/A in TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:25,583 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8be8843d5bbff3d4194de143513dc64b/B in TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:25,583 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/6564c1724fc14cd8be95892afc2e6121, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/29fcdd96a0c148959e87a373fb7599df, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/90f24717e0e5434b8d8a65a8a5c5928f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/a4d1a40f2b1a4860bba3ceef568b05e3, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/9c867c1d12b5449bb81be16cbc6d75e0] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp, totalSize=133.6 K 2023-04-08 02:55:25,583 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/7c073e081ccf417182b58773a647a7b0, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/13eed4c94b294bcea7b242e6541cc153, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/9f04d1f1f6824ae0ae0fcaaadb7a5b9f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/34eb14d15d0341b99b8c2ad0ab743cf1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/665807a0694a4cd49c52de79bfb59972] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp, totalSize=56.7 K 2023-04-08 02:55:25,584 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 6564c1724fc14cd8be95892afc2e6121, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680922521065 2023-04-08 02:55:25,584 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 7c073e081ccf417182b58773a647a7b0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680922521065 2023-04-08 02:55:25,584 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 29fcdd96a0c148959e87a373fb7599df, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680922521458 2023-04-08 02:55:25,584 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 13eed4c94b294bcea7b242e6541cc153, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680922521466 2023-04-08 02:55:25,585 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 90f24717e0e5434b8d8a65a8a5c5928f, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680922521495 2023-04-08 02:55:25,585 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 9f04d1f1f6824ae0ae0fcaaadb7a5b9f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680922521495 2023-04-08 02:55:25,585 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting a4d1a40f2b1a4860bba3ceef568b05e3, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1680922523774 2023-04-08 02:55:25,585 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 34eb14d15d0341b99b8c2ad0ab743cf1, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1680922523774 2023-04-08 02:55:25,586 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 9c867c1d12b5449bb81be16cbc6d75e0, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680922524667 2023-04-08 02:55:25,586 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 665807a0694a4cd49c52de79bfb59972, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680922524667 2023-04-08 02:55:25,595 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 8be8843d5bbff3d4194de143513dc64b#B#compaction#392 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:25,606 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8be8843d5bbff3d4194de143513dc64b#A#compaction#393 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:25,609 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:25,610 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:25,612 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:25,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:25,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:25,613 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:25,613 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:25,613 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:25,613 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:25,613 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,613 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:25,616 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:25,616 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:25,616 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:25,616 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:25,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:25,616 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:25,617 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:25,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:25,617 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:25,617 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:25,620 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:25,620 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:25,620 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:25,620 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:25,620 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:25,620 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:25,620 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:25,620 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:25,620 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:25,620 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8be8843d5bbff3d4194de143513dc64b 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-08 02:55:25,621 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=A 2023-04-08 02:55:25,621 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:25,621 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=B 2023-04-08 02:55:25,621 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:25,621 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=C 2023-04-08 02:55:25,621 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:25,915 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:25,915 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:26,011 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/3a5f5d43581d41dea85b752f8a3de420 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/3a5f5d43581d41dea85b752f8a3de420 2023-04-08 02:55:26,015 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:26,015 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 326 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47700 deadline: 1680922586007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,047 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8be8843d5bbff3d4194de143513dc64b/B of 8be8843d5bbff3d4194de143513dc64b into 3a5f5d43581d41dea85b752f8a3de420(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:26,047 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:26,047 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., storeName=8be8843d5bbff3d4194de143513dc64b/B, priority=11, startTime=1680922525582; duration=0sec 2023-04-08 02:55:26,047 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:26,047 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 02:55:26,048 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408fa8a5a97150246e48038011d5bea0d46 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408fa8a5a97150246e48038011d5bea0d46 2023-04-08 02:55:26,049 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58076 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 02:55:26,050 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 8be8843d5bbff3d4194de143513dc64b/C is initiating minor compaction (all files) 2023-04-08 02:55:26,050 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8be8843d5bbff3d4194de143513dc64b/C in TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:26,050 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/d1c075e5002c441a99d6cb840a85bda2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/48f755102cc54276b4e4ddfee5905b78, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/d39e8f81335341d984d2ee464f734285, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/916edf6e8df34fd09b14d9400b7c61cd, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/276ce84520c1494396f2c6b363e07e26] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp, totalSize=56.7 K 2023-04-08 02:55:26,050 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting d1c075e5002c441a99d6cb840a85bda2, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=130, earliestPutTs=1680922521065 2023-04-08 02:55:26,050 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 48f755102cc54276b4e4ddfee5905b78, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=147, earliestPutTs=1680922521466 2023-04-08 02:55:26,051 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting d39e8f81335341d984d2ee464f734285, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=165, earliestPutTs=1680922521495 2023-04-08 02:55:26,052 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 916edf6e8df34fd09b14d9400b7c61cd, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=173, earliestPutTs=1680922523774 2023-04-08 02:55:26,052 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/c3363193596a40cd814ed4c09165b120 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/c3363193596a40cd814ed4c09165b120 2023-04-08 02:55:26,052 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 276ce84520c1494396f2c6b363e07e26, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680922524667 2023-04-08 02:55:26,056 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=340, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/709db83f505f48a69f04775a5e1b924b 2023-04-08 02:55:26,059 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8be8843d5bbff3d4194de143513dc64b/A of 8be8843d5bbff3d4194de143513dc64b into c3363193596a40cd814ed4c09165b120(size=25.8 K), total size for store is 25.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:26,059 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:26,059 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., storeName=8be8843d5bbff3d4194de143513dc64b/A, priority=11, startTime=1680922525582; duration=0sec 2023-04-08 02:55:26,060 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:26,066 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 8be8843d5bbff3d4194de143513dc64b#C#compaction#396 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:26,080 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/334d55cdf41b4592aab0d7cd2c55ba80 2023-04-08 02:55:26,088 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/e120906470bc4bc3894269541f1c5670 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/e120906470bc4bc3894269541f1c5670 2023-04-08 02:55:26,090 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=340 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/4a472bf1aae8489fa078ebba0ff0f39e 2023-04-08 02:55:26,093 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 8be8843d5bbff3d4194de143513dc64b/C of 8be8843d5bbff3d4194de143513dc64b into e120906470bc4bc3894269541f1c5670(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:26,094 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:26,094 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., storeName=8be8843d5bbff3d4194de143513dc64b/C, priority=11, startTime=1680922525582; duration=0sec 2023-04-08 02:55:26,094 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:26,096 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/709db83f505f48a69f04775a5e1b924b as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/709db83f505f48a69f04775a5e1b924b 2023-04-08 02:55:26,101 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/709db83f505f48a69f04775a5e1b924b, entries=150, sequenceid=340, filesize=25.4 K 2023-04-08 02:55:26,102 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/334d55cdf41b4592aab0d7cd2c55ba80 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/334d55cdf41b4592aab0d7cd2c55ba80 2023-04-08 02:55:26,105 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/334d55cdf41b4592aab0d7cd2c55ba80, entries=150, sequenceid=340, filesize=11.9 K 2023-04-08 02:55:26,105 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/4a472bf1aae8489fa078ebba0ff0f39e as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/4a472bf1aae8489fa078ebba0ff0f39e 2023-04-08 02:55:26,109 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/4a472bf1aae8489fa078ebba0ff0f39e, entries=150, sequenceid=340, filesize=11.9 K 2023-04-08 02:55:26,109 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 8be8843d5bbff3d4194de143513dc64b in 489ms, sequenceid=340, compaction requested=false 2023-04-08 02:55:26,110 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:26,110 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:26,110 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:26,110 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:26,110 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,110 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:26,110 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:26,111 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,111 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:26,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:26,112 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,112 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:26,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:26,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:26,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:26,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:26,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:26,113 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9caf082[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:26,113 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:26,113 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,115 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,115 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:26,115 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:26,115 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:26,117 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:26,117 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:26,117 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,117 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:26,118 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:26,118 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:26,118 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:26,118 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:26,118 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:26,118 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:26,118 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:26,120 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,120 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:26,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,120 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:26,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:26,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:26,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:26,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:26,121 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:26,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:26,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:26,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,122 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:26,123 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8be8843d5bbff3d4194de143513dc64b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-08 02:55:26,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:26,123 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,123 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=A 2023-04-08 02:55:26,123 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:26,123 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=B 2023-04-08 02:55:26,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:26,123 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:26,123 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=C 2023-04-08 02:55:26,123 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:26,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,125 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,125 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:26,125 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:26,125 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,125 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:26,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:26,125 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:26,126 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:26,125 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,126 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,126 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,126 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:26,125 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:26,126 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:26,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:26,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:26,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:26,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:26,135 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040870560bd0651949dfbd3ca5eb9cfc94d1 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040870560bd0651949dfbd3ca5eb9cfc94d1 2023-04-08 02:55:26,165 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:26,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 339 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47700 deadline: 1680922586164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,226 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:26,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:26,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:26,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:26,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:26,229 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:26,229 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:26,229 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:26,229 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:26,229 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,229 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:26,231 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,231 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:26,231 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:26,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:26,231 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,231 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:26,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:26,231 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:26,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:26,232 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:26,232 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:26,232 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:26,232 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:26,232 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:26,232 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:26,233 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:26,233 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:26,233 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:26,233 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:26,233 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:26,233 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:26,233 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:26,233 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:26,233 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,233 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:26,233 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:26,234 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,234 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:26,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:26,234 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,234 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:26,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:26,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:26,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:26,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:26,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:26,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39fb9685[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:26,235 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:26,236 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,237 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,237 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:26,237 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:26,237 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:26,238 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,238 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:26,239 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,239 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:26,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:26,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:26,239 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:26,239 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:26,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:26,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:26,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:26,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:26,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:26,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:26,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:26,241 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:26,241 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:26,241 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:26,241 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:26,241 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:26,242 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,242 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:26,242 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,242 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:26,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:26,243 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:26,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:26,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:26,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:26,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:26,244 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:26,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:26,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:26,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,245 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,246 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:26,246 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:26,246 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:26,246 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:26,246 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:26,246 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:26,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:26,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:26,246 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,246 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,246 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,246 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:26,246 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:26,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:26,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:26,246 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:26,273 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:26,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 341 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47700 deadline: 1680922586266, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,346 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:26,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:26,348 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:26,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:26,349 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:26,349 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:26,349 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:26,350 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:26,350 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:26,350 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,350 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:26,352 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,352 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:26,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:26,352 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,352 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:26,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:26,353 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:26,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:26,353 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:26,353 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:26,353 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:26,353 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:26,353 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:26,353 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:26,353 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:26,353 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:26,353 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:26,354 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:26,354 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:26,354 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:26,354 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:26,354 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:26,354 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,354 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:26,354 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:26,355 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,355 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:26,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:26,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:26,356 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,356 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:26,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:26,358 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@442c8f6d[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:26,358 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:26,358 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,359 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,359 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,359 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:26,359 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:26,359 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:26,359 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,359 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:26,360 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:26,360 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:26,360 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,360 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:26,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:26,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:26,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:26,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:26,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:26,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:26,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:26,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:26,362 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:26,362 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:26,362 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:26,362 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:26,362 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:26,362 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:26,364 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,364 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,364 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:26,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,365 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:26,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:26,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:26,366 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:26,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:26,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,370 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:26,370 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:26,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:26,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:26,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:26,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:26,376 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,376 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:26,376 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:26,376 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:26,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:26,376 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:26,376 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,376 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,376 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,376 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,376 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:26,376 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:26,476 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:26,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47700 deadline: 1680922586474, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,477 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:26,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:26,478 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:26,479 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:26,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:26,480 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:26,480 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:26,480 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:26,480 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:26,480 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,480 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:26,481 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:26,481 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:26,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:26,482 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,482 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:26,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:26,482 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:26,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:26,483 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:26,483 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:26,483 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:26,483 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:26,483 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:26,483 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:26,483 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:26,483 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:26,483 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:26,483 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:26,483 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:26,483 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:26,483 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:26,483 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:26,483 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,483 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:26,483 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:26,486 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,486 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:26,486 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,486 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:26,486 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:26,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,487 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:26,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:26,487 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@16bb62a1[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:26,487 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:26,487 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,488 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,488 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,488 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,488 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,488 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:26,488 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:26,488 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:26,488 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:26,489 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:26,489 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,489 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:26,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:26,490 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:26,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:26,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:26,492 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:26,492 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:26,492 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:26,492 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:26,492 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:26,494 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,494 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,494 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:26,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:26,494 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:26,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:26,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:26,495 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:26,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:26,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,498 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,498 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:26,498 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:26,498 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:26,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:26,499 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:26,498 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:26,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:26,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:26,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:26,498 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,499 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,499 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:26,499 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,499 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:26,499 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:26,499 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:26,499 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:26,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:26,541 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=362, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/559173e8f9c9420cb1e9614c03efe583 2023-04-08 02:55:26,554 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/ea64c9b0e25043088feeb44e587906e0 2023-04-08 02:55:26,572 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=362 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/53cba284dfd14a169a5ebb48e62d1a6b 2023-04-08 02:55:26,576 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/559173e8f9c9420cb1e9614c03efe583 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/559173e8f9c9420cb1e9614c03efe583 2023-04-08 02:55:26,579 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/559173e8f9c9420cb1e9614c03efe583, entries=200, sequenceid=362, filesize=32.3 K 2023-04-08 02:55:26,580 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/ea64c9b0e25043088feeb44e587906e0 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/ea64c9b0e25043088feeb44e587906e0 2023-04-08 02:55:26,583 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/ea64c9b0e25043088feeb44e587906e0, entries=150, sequenceid=362, filesize=11.9 K 2023-04-08 02:55:26,584 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/53cba284dfd14a169a5ebb48e62d1a6b as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/53cba284dfd14a169a5ebb48e62d1a6b 2023-04-08 02:55:26,587 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/53cba284dfd14a169a5ebb48e62d1a6b, entries=150, sequenceid=362, filesize=11.9 K 2023-04-08 02:55:26,587 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 8be8843d5bbff3d4194de143513dc64b in 464ms, sequenceid=362, compaction requested=true 2023-04-08 02:55:26,587 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:26,588 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:26,588 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:55:26,588 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:55:26,588 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:26,588 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:26,591 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 85588 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:55:26,591 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:55:26,591 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 8be8843d5bbff3d4194de143513dc64b/A is initiating minor compaction (all files) 2023-04-08 02:55:26,592 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 8be8843d5bbff3d4194de143513dc64b/B is initiating minor compaction (all files) 2023-04-08 02:55:26,592 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8be8843d5bbff3d4194de143513dc64b/A in TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:26,592 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8be8843d5bbff3d4194de143513dc64b/B in TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:26,592 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/c3363193596a40cd814ed4c09165b120, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/709db83f505f48a69f04775a5e1b924b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/559173e8f9c9420cb1e9614c03efe583] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp, totalSize=83.6 K 2023-04-08 02:55:26,592 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/3a5f5d43581d41dea85b752f8a3de420, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/334d55cdf41b4592aab0d7cd2c55ba80, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/ea64c9b0e25043088feeb44e587906e0] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp, totalSize=36.1 K 2023-04-08 02:55:26,592 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting c3363193596a40cd814ed4c09165b120, keycount=150, bloomtype=ROW, size=25.8 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680922524667 2023-04-08 02:55:26,592 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 3a5f5d43581d41dea85b752f8a3de420, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680922524667 2023-04-08 02:55:26,593 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 709db83f505f48a69f04775a5e1b924b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1680922524741 2023-04-08 02:55:26,594 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 334d55cdf41b4592aab0d7cd2c55ba80, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1680922524741 2023-04-08 02:55:26,594 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 559173e8f9c9420cb1e9614c03efe583, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1680922525990 2023-04-08 02:55:26,594 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting ea64c9b0e25043088feeb44e587906e0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1680922525997 2023-04-08 02:55:26,599 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:26,600 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:26,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:26,607 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 8be8843d5bbff3d4194de143513dc64b#B#compaction#402 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:26,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:26,608 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:26,608 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:26,608 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:26,608 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:26,608 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:26,609 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,609 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:26,610 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:26,610 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,610 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:26,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:26,610 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,610 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:26,610 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:26,611 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:26,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:26,611 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:26,611 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:26,617 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:26,617 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:26,617 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:26,617 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:26,617 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:26,621 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8be8843d5bbff3d4194de143513dc64b#A#compaction#401 average throughput is 0.90 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:26,623 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:26,624 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:26,624 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:26,624 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:26,624 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8be8843d5bbff3d4194de143513dc64b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-08 02:55:26,624 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=A 2023-04-08 02:55:26,624 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:26,624 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=B 2023-04-08 02:55:26,624 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:26,624 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=C 2023-04-08 02:55:26,624 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:26,635 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/676c721ebef0402489a8ad65f0d6c512 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/676c721ebef0402489a8ad65f0d6c512 2023-04-08 02:55:26,639 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8be8843d5bbff3d4194de143513dc64b/B of 8be8843d5bbff3d4194de143513dc64b into 676c721ebef0402489a8ad65f0d6c512(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:26,639 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:26,639 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., storeName=8be8843d5bbff3d4194de143513dc64b/B, priority=13, startTime=1680922526588; duration=0sec 2023-04-08 02:55:26,639 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:26,639 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:55:26,640 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36968 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:55:26,640 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 8be8843d5bbff3d4194de143513dc64b/C is initiating minor compaction (all files) 2023-04-08 02:55:26,640 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8be8843d5bbff3d4194de143513dc64b/C in TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:26,640 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/e120906470bc4bc3894269541f1c5670, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/4a472bf1aae8489fa078ebba0ff0f39e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/53cba284dfd14a169a5ebb48e62d1a6b] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp, totalSize=36.1 K 2023-04-08 02:55:26,640 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting e120906470bc4bc3894269541f1c5670, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=326, earliestPutTs=1680922524667 2023-04-08 02:55:26,641 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 4a472bf1aae8489fa078ebba0ff0f39e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=340, earliestPutTs=1680922524741 2023-04-08 02:55:26,641 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 53cba284dfd14a169a5ebb48e62d1a6b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1680922525997 2023-04-08 02:55:26,649 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 8be8843d5bbff3d4194de143513dc64b#C#compaction#404 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:26,656 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304083ada5ac2e61449e88a2b66cb905b7317 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304083ada5ac2e61449e88a2b66cb905b7317 2023-04-08 02:55:26,668 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/11fc7126a9e24a288004054d40f51678 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/11fc7126a9e24a288004054d40f51678 2023-04-08 02:55:26,673 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8be8843d5bbff3d4194de143513dc64b/C of 8be8843d5bbff3d4194de143513dc64b into 11fc7126a9e24a288004054d40f51678(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:26,673 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:26,673 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., storeName=8be8843d5bbff3d4194de143513dc64b/C, priority=13, startTime=1680922526588; duration=0sec 2023-04-08 02:55:26,673 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:26,675 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:26,675 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:26,710 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:26,711 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922586705, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,712 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:26,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922586708, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,713 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:26,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47734 deadline: 1680922586710, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,718 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:26,718 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922586711, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,782 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:26,782 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47700 deadline: 1680922586778, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,814 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:26,814 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922586812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,815 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:26,815 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922586813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,818 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:26,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47734 deadline: 1680922586814, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:26,820 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:26,820 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922586819, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,018 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:27,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922587015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,020 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:27,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922587017, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,023 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:27,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47734 deadline: 1680922587020, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,024 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:27,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922587021, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,061 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=375, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/0ee17eaa4ffd4058930b9743fca1d618 2023-04-08 02:55:27,287 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:27,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47700 deadline: 1680922587284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,323 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:27,323 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922587320, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,325 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:27,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922587323, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:27,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47734 deadline: 1680922587325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:27,328 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922587326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,446 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/582d39ea57434b568e074014f989055d as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/582d39ea57434b568e074014f989055d 2023-04-08 02:55:27,450 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8be8843d5bbff3d4194de143513dc64b/A of 8be8843d5bbff3d4194de143513dc64b into 582d39ea57434b568e074014f989055d(size=25.9 K), total size for store is 25.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:27,450 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:27,450 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., storeName=8be8843d5bbff3d4194de143513dc64b/A, priority=13, startTime=1680922526588; duration=0sec 2023-04-08 02:55:27,451 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:27,473 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/3530bf85a80d4128b435cd58e0634c82 2023-04-08 02:55:27,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:27,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922587826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,831 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:27,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922587829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,833 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:27,833 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922587831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,833 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:27,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47734 deadline: 1680922587831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,883 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=375 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/faed3aa5fc7d40d0aea35d921a06a751 2023-04-08 02:55:27,886 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/0ee17eaa4ffd4058930b9743fca1d618 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/0ee17eaa4ffd4058930b9743fca1d618 2023-04-08 02:55:27,890 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/0ee17eaa4ffd4058930b9743fca1d618, entries=150, sequenceid=375, filesize=25.4 K 2023-04-08 02:55:27,891 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/3530bf85a80d4128b435cd58e0634c82 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/3530bf85a80d4128b435cd58e0634c82 2023-04-08 02:55:27,894 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/3530bf85a80d4128b435cd58e0634c82, entries=150, sequenceid=375, filesize=11.9 K 2023-04-08 02:55:27,895 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/faed3aa5fc7d40d0aea35d921a06a751 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/faed3aa5fc7d40d0aea35d921a06a751 2023-04-08 02:55:27,899 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/faed3aa5fc7d40d0aea35d921a06a751, entries=150, sequenceid=375, filesize=11.9 K 2023-04-08 02:55:27,900 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 8be8843d5bbff3d4194de143513dc64b in 1276ms, sequenceid=375, compaction requested=false 2023-04-08 02:55:27,900 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:27,900 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:27,900 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:27,900 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:27,900 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,900 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:27,900 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:27,901 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:27,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:27,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:27,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:27,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:27,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:27,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:27,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:27,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:27,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:27,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48abde14[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:27,903 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:27,903 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:27,904 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:27,904 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:27,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:27,904 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:27,905 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:27,905 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:27,905 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,905 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:27,907 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:27,907 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,907 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:27,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,907 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:27,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:27,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:27,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:27,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:27,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:27,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:27,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:27,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:27,909 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:27,909 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:27,910 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:27,910 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:27,910 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:27,910 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:27,911 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:27,911 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:27,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:27,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:27,911 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:27,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:27,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:27,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:27,911 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:27,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:27,911 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:27,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:27,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:27,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:27,913 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:27,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:27,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:27,915 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:27,915 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:27,915 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:27,915 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:27,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:27,915 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:27,915 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:27,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:27,915 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:27,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:27,915 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:27,916 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:27,916 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:27,916 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:27,915 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:27,916 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:27,916 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:28,016 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:28,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:28,018 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:28,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:28,019 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:28,019 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:28,019 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:28,019 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:28,019 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:28,020 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:28,020 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:28,021 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:28,021 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:28,021 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:28,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:28,021 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:28,021 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:28,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:28,022 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:28,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:28,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:28,022 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:28,022 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:28,022 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:28,022 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:28,022 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:28,022 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:28,023 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:28,023 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:28,023 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:28,023 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:28,023 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8be8843d5bbff3d4194de143513dc64b 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-08 02:55:28,023 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=A 2023-04-08 02:55:28,023 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:28,023 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=B 2023-04-08 02:55:28,023 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:28,024 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=C 2023-04-08 02:55:28,024 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:28,040 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304082ead7753fecf4acf812f1e20fb0a171c to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304082ead7753fecf4acf812f1e20fb0a171c 2023-04-08 02:55:28,292 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:28,292 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:28,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:28,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47700 deadline: 1680922588330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:28,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:28,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47700 deadline: 1680922588435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:28,446 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=398, memsize=38.0 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/44bb786324104965b47c2a90b9dd854a 2023-04-08 02:55:28,458 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=398 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/ff9513a72e30469d94ec7febf405af5c 2023-04-08 02:55:28,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:28,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47700 deadline: 1680922588639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:28,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:28,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922588834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:28,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:28,838 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922588835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:28,843 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:28,844 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922588840, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:28,845 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:28,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47734 deadline: 1680922588843, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:28,868 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=398 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/1a041131b7cf419ab4d7d41cd8ef3b8c 2023-04-08 02:55:28,872 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/44bb786324104965b47c2a90b9dd854a as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/44bb786324104965b47c2a90b9dd854a 2023-04-08 02:55:28,875 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/44bb786324104965b47c2a90b9dd854a, entries=150, sequenceid=398, filesize=25.4 K 2023-04-08 02:55:28,876 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/ff9513a72e30469d94ec7febf405af5c as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/ff9513a72e30469d94ec7febf405af5c 2023-04-08 02:55:28,879 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/ff9513a72e30469d94ec7febf405af5c, entries=150, sequenceid=398, filesize=11.9 K 2023-04-08 02:55:28,880 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/1a041131b7cf419ab4d7d41cd8ef3b8c as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/1a041131b7cf419ab4d7d41cd8ef3b8c 2023-04-08 02:55:28,883 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/1a041131b7cf419ab4d7d41cd8ef3b8c, entries=150, sequenceid=398, filesize=11.9 K 2023-04-08 02:55:28,883 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 8be8843d5bbff3d4194de143513dc64b in 860ms, sequenceid=398, compaction requested=true 2023-04-08 02:55:28,883 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:28,883 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:28,883 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:28,883 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:28,883 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:28,883 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:28,883 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:28,885 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:28,885 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:28,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:28,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:28,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:28,885 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:28,885 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:28,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:28,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:28,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:28,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:28,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:28,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:28,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d01363a[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:28,886 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:28,886 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:28,887 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:28,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:28,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:28,887 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:28,887 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:28,888 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:28,888 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:28,888 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:28,889 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:28,889 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:28,889 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:28,889 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:28,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:28,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:28,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:28,891 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:28,891 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:28,891 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:28,891 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:28,891 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:28,891 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:28,894 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:28,894 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:28,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:28,894 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:28,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:28,894 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:28,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:28,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:28,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:28,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:28,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:28,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:28,895 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:28,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:28,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:28,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:28,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:28,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:28,897 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:28,897 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:28,897 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:28,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:28,898 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:28,897 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:28,897 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:28,898 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:28,898 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:28,898 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:28,897 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:28,898 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:28,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:28,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:28,898 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:28,898 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:28,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:28,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:28,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:28,951 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8be8843d5bbff3d4194de143513dc64b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-08 02:55:28,951 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=A 2023-04-08 02:55:28,951 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:28,951 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=B 2023-04-08 02:55:28,951 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:28,951 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=C 2023-04-08 02:55:28,951 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:28,998 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:28,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:29,000 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:29,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:29,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:29,002 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:29,002 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:29,002 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:29,002 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:29,002 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,003 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,004 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:29,004 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:29,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:29,004 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,004 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:29,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,004 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:29,005 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,005 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:29,005 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:29,005 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:29,005 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:29,005 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:29,005 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:29,005 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:29,005 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:29,005 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:29,005 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:29,006 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:29,006 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:29,006 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:29,006 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,006 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:29,006 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:29,010 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,010 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:29,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:29,011 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,011 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:29,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:29,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:29,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:29,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:29,012 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@d768310[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:29,012 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:29,012 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,013 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,013 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,013 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:29,014 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:29,014 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:29,014 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,014 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:29,015 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,015 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:29,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:29,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:29,015 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:29,015 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:29,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:29,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:29,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:29,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:29,017 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:29,017 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:29,017 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:29,017 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:29,017 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:29,017 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:29,019 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,019 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,019 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:29,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:29,019 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:29,019 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:29,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:29,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,020 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:29,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,021 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,023 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,023 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,023 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,023 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,023 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,023 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:29,023 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:29,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:29,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:29,023 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:29,023 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:29,023 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,023 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:29,024 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:29,024 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:29,024 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:29,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:29,062 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:29,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 381 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47700 deadline: 1680922589056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,124 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:29,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:29,126 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:29,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:29,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:29,127 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:29,127 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:29,127 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:29,127 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:29,127 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,127 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,129 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:29,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:29,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:29,129 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,129 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,129 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:29,129 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:29,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,130 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:29,130 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:29,130 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:29,130 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:29,130 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:29,130 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:29,130 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:29,130 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:29,131 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:29,131 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:29,131 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:29,131 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:29,131 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:29,131 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:29,131 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,131 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:29,131 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:29,132 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,132 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:29,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:29,133 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,133 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:29,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:29,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:29,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:29,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:29,134 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@40954675[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:29,134 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:29,134 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,135 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,135 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:29,135 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:29,135 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,135 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:29,135 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:29,138 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:29,138 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:29,138 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,138 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:29,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:29,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:29,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:29,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:29,138 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,139 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:29,140 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:29,140 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:29,140 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:29,140 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:29,140 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:29,140 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:29,141 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,141 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,141 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,141 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:29,141 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:29,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:29,142 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:29,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:29,142 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:29,142 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:29,143 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,143 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:29,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,145 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,145 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,145 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,145 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,145 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:29,145 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:29,145 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:29,145 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,145 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:29,145 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,145 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:29,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:29,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:29,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:29,145 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:29,145 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:29,146 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:29,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:29,165 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:29,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 383 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47700 deadline: 1680922589164, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,246 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:29,246 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:29,248 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:29,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:29,249 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:29,249 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:29,250 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:29,250 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:29,250 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:29,250 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,250 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,251 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:29,251 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,251 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:29,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:29,252 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,252 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:29,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,253 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:29,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,253 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:29,255 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:29,255 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:29,255 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:29,256 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:29,256 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:29,256 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:29,256 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:29,256 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:29,256 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:29,256 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:29,256 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:29,256 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:29,256 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:29,256 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,256 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:29,256 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:29,262 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,262 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:29,262 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:29,263 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:29,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:29,264 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:29,264 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56586b0d[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:29,264 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,265 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,265 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,265 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,265 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:29,265 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:29,265 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:29,265 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,265 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:29,266 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:29,267 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:29,267 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:29,267 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:29,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,269 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:29,269 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:29,269 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:29,269 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:29,269 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:29,269 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:29,269 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:29,271 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,271 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,271 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:29,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:29,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:29,271 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:29,272 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,272 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:29,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:29,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,272 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,273 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:29,273 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,276 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:29,276 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:29,276 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:29,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:29,276 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:29,276 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:29,276 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:29,276 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,277 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,277 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,277 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:29,277 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:29,277 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,277 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:29,277 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:29,277 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,277 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,362 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040894138417d1994606947eb94b4e3a8444 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040894138417d1994606947eb94b4e3a8444 2023-04-08 02:55:29,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:29,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 385 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47700 deadline: 1680922589366, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,377 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:29,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:29,380 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:29,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:29,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:29,381 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:29,381 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:29,381 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:29,381 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:29,381 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,381 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,383 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:29,383 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:29,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:29,383 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,383 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:29,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:29,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,384 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:29,384 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:29,384 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:29,384 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:29,384 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:29,384 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:29,384 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:29,384 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:29,384 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:29,384 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:29,384 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:29,385 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:29,385 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:29,385 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:29,385 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,385 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:29,385 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:29,386 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:29,386 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:29,387 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,387 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:29,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:29,388 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:29,388 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@72cc271a[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:29,388 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,389 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,389 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,389 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:29,389 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,389 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:29,389 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:29,389 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:29,394 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,394 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:29,394 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:29,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:29,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:29,394 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:29,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:29,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:29,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:29,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:29,396 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:29,396 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:29,396 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:29,396 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:29,397 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:29,397 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:29,398 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:29,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:29,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:29,399 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,399 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,399 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:29,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:29,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:29,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:29,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,400 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:29,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,402 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,402 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,402 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:29,402 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,402 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:29,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:29,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:29,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:29,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:29,402 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,402 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,402 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,402 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,402 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:29,402 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:29,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:29,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:29,403 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:29,503 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:29,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:29,505 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:29,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:29,506 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:29,506 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:29,506 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:29,507 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:29,507 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:29,507 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,507 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,508 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,508 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:29,508 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:29,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:29,508 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,508 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:29,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,509 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:29,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,509 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:29,510 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:29,510 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:29,510 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:29,510 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:29,510 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:29,510 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:29,510 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:29,510 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:29,510 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:29,510 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:29,510 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:29,510 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:29,510 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:29,510 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,510 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:29,511 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:29,513 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,513 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:29,513 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,513 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:29,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:29,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:29,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:29,515 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b1b5450[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:29,515 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:29,515 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,516 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,516 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:29,517 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,517 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:29,517 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:29,517 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:29,519 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,519 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:29,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:29,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:29,519 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:29,519 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:29,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:29,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:29,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:29,521 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:29,521 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:29,521 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:29,521 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:29,521 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:29,521 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:29,523 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,523 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:29,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,523 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:29,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:29,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:29,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:29,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:29,524 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:29,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,526 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,530 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:29,530 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:29,530 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:29,530 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:29,530 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,530 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,530 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,530 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,530 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,530 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,531 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:29,531 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:29,536 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:29,536 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:29,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:29,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:29,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:29,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:29,631 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:29,631 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:29,633 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:29,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:29,634 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:29,634 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:29,634 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:29,634 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:29,634 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:29,635 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,635 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,636 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,636 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:29,636 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:29,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:29,636 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,636 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:29,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,637 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:29,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,637 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:29,637 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:29,637 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:29,637 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:29,637 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:29,637 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:29,637 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:29,638 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:29,638 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:29,638 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:29,638 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:29,638 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:29,638 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:29,638 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:29,638 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,638 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:29,638 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:29,639 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,639 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:29,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:29,640 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,640 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:29,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:29,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:29,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:29,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:29,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@215715a8[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:29,641 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:29,641 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,642 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,643 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:29,643 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:29,643 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:29,643 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,643 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:29,645 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:29,645 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:29,645 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,645 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:29,649 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:29,649 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:29,649 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:29,649 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:29,649 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:29,649 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:29,650 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,650 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:29,650 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,650 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:29,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,651 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:29,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:29,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:29,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:29,652 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:29,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,654 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,654 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,654 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,654 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,654 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,654 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:29,654 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:29,654 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,654 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:29,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:29,654 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:29,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:29,654 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:29,654 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:29,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:29,655 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:29,655 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:29,674 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:29,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 387 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47700 deadline: 1680922589670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,755 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:29,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:29,757 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:29,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:29,758 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:29,758 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:29,758 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:29,758 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:29,758 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:29,758 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,758 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,760 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:29,760 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:29,760 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,760 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,760 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:29,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,761 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:29,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,761 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:29,761 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:29,761 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:29,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:29,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:29,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:29,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:29,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:29,762 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:29,762 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:29,762 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:29,762 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:29,762 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:29,762 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:29,762 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,762 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:29,762 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:29,764 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,764 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:29,764 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,764 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:29,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:29,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:29,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:29,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:29,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3442028c[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:29,766 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:29,766 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,767 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=411, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/18c7f50b3bd243f2a74ac1ec8f70ef4e 2023-04-08 02:55:29,767 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,767 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,767 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:29,767 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,767 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:29,767 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:29,767 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:29,769 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,769 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:29,769 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:29,769 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:29,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:29,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:29,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:29,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:29,774 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:29,774 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:29,774 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:29,775 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:29,775 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:29,785 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,785 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,785 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:29,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:29,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:29,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,785 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:29,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:29,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:29,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:29,786 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:29,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,788 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,788 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:29,788 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:29,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:29,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:29,788 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:29,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:29,788 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,788 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,788 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,788 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:29,788 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:29,788 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:29,788 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,788 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:29,788 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,789 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,888 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:29,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:29,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:29,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:29,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:29,892 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:29,892 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:29,892 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:29,892 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:29,892 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,892 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,894 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:29,894 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:29,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:29,894 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,894 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:29,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,894 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:29,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,895 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:29,895 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:29,895 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:29,895 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:29,895 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:29,895 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:29,896 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:29,896 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:29,896 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:29,896 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:29,896 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:29,896 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:29,896 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:29,896 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:29,896 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,896 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:29,896 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:29,898 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,898 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:29,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:29,898 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,899 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:29,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:29,900 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@272517e9[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:29,900 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:29,900 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,901 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,901 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,901 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:29,901 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:29,901 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:29,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:29,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:29,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:29,903 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,903 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:29,903 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:29,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:29,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:29,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:29,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:29,905 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:29,905 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:29,905 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:29,905 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:29,905 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:29,906 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,906 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,906 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,906 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:29,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:29,907 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:29,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,908 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:29,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:29,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,916 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:29,916 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:29,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:29,917 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:29,917 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,917 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:29,917 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:29,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:29,917 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:29,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:29,917 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:29,917 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:29,917 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,917 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:29,917 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:29,917 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:29,917 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:30,017 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:30,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:30,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:30,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:30,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:30,022 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:30,022 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:30,022 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:30,022 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:30,022 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:30,022 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:30,023 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,023 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:30,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:30,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:30,024 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,024 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:30,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:30,024 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:30,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:30,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:30,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:30,025 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:30,025 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:30,025 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:30,025 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:30,025 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:30,025 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:30,025 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:30,027 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,027 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:30,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:30,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:30,027 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:30,027 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:30,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:30,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:30,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:30,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:30,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:30,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@360d7e89[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:30,029 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:30,029 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:30,030 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:30,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:30,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:30,030 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:30,030 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:30,030 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,030 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:30,030 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:30,032 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,032 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:30,032 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:30,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:30,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:30,032 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:30,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:30,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:30,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:30,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:30,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:30,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:30,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:30,034 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:30,034 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:30,034 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:30,034 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:30,034 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:30,036 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:30,036 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:30,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:30,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:30,036 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:30,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:30,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:30,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:30,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:30,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:30,036 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:30,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:30,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:30,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:30,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:30,037 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:30,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:30,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,039 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:30,039 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:30,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:30,039 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:30,039 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:30,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:30,039 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:30,039 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:30,039 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,039 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:30,039 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:30,039 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:30,040 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:30,040 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:30,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:30,039 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:30,040 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:30,040 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:30,140 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:30,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:30,142 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:30,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:30,143 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:30,143 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:30,143 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:30,143 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:30,143 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:30,143 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:30,144 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:30,145 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,145 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:30,145 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:30,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:30,145 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,145 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:30,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:30,145 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:30,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:30,146 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:30,146 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:30,146 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:30,146 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:30,146 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:30,146 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:30,146 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:30,146 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:30,146 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:30,146 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:30,146 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:30,146 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:30,146 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:30,146 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:30,146 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:30,147 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,147 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:30,147 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:30,152 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:30,152 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:30,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:30,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:30,153 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:30,153 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:30,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:30,153 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:30,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:30,154 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:30,155 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2feec7b5[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:30,155 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:30,155 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:30,156 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:30,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:30,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:30,156 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:30,156 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,156 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:30,156 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:30,156 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:30,157 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:30,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:30,157 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:30,157 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:30,157 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:30,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:30,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:30,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:30,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:30,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:30,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:30,159 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:30,159 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:30,159 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:30,159 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:30,159 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:30,159 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:30,161 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:30,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:30,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:30,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:30,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:30,161 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:30,161 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:30,161 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:30,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:30,162 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:30,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,162 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:30,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:30,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,168 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:30,168 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:30,168 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:30,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:30,169 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:30,168 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:30,168 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:30,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:30,169 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:30,169 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:30,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:30,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:30,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:30,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:30,177 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/9beeb3b7e4074af9867ac03aab10856a 2023-04-08 02:55:30,182 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:30,182 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 389 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47700 deadline: 1680922590177, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,186 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=411 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/be16bc1766f242fda7f1920507d21fa8 2023-04-08 02:55:30,189 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/18c7f50b3bd243f2a74ac1ec8f70ef4e as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/18c7f50b3bd243f2a74ac1ec8f70ef4e 2023-04-08 02:55:30,192 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/18c7f50b3bd243f2a74ac1ec8f70ef4e, entries=200, sequenceid=411, filesize=32.3 K 2023-04-08 02:55:30,193 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/9beeb3b7e4074af9867ac03aab10856a as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/9beeb3b7e4074af9867ac03aab10856a 2023-04-08 02:55:30,196 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/9beeb3b7e4074af9867ac03aab10856a, entries=150, sequenceid=411, filesize=11.9 K 2023-04-08 02:55:30,197 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/be16bc1766f242fda7f1920507d21fa8 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/be16bc1766f242fda7f1920507d21fa8 2023-04-08 02:55:30,200 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/be16bc1766f242fda7f1920507d21fa8, entries=150, sequenceid=411, filesize=11.9 K 2023-04-08 02:55:30,201 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 8be8843d5bbff3d4194de143513dc64b in 1250ms, sequenceid=411, compaction requested=true 2023-04-08 02:55:30,201 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:30,201 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:30,201 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:55:30,201 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:30,201 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-08 02:55:30,201 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:55:30,202 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 111736 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:55:30,203 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 8be8843d5bbff3d4194de143513dc64b/A is initiating minor compaction (all files) 2023-04-08 02:55:30,203 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8be8843d5bbff3d4194de143513dc64b/A in TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:30,203 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:55:30,203 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/582d39ea57434b568e074014f989055d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/0ee17eaa4ffd4058930b9743fca1d618, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/44bb786324104965b47c2a90b9dd854a, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/18c7f50b3bd243f2a74ac1ec8f70ef4e] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp, totalSize=109.1 K 2023-04-08 02:55:30,203 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 8be8843d5bbff3d4194de143513dc64b/B is initiating minor compaction (all files) 2023-04-08 02:55:30,203 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8be8843d5bbff3d4194de143513dc64b/B in TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:30,203 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/676c721ebef0402489a8ad65f0d6c512, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/3530bf85a80d4128b435cd58e0634c82, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/ff9513a72e30469d94ec7febf405af5c, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/9beeb3b7e4074af9867ac03aab10856a] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp, totalSize=48.1 K 2023-04-08 02:55:30,203 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 582d39ea57434b568e074014f989055d, keycount=150, bloomtype=ROW, size=25.9 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1680922525997 2023-04-08 02:55:30,204 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 0ee17eaa4ffd4058930b9743fca1d618, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1680922526148 2023-04-08 02:55:30,204 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 676c721ebef0402489a8ad65f0d6c512, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1680922525997 2023-04-08 02:55:30,204 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 44bb786324104965b47c2a90b9dd854a, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1680922526699 2023-04-08 02:55:30,204 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 3530bf85a80d4128b435cd58e0634c82, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1680922526148 2023-04-08 02:55:30,206 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting ff9513a72e30469d94ec7febf405af5c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1680922526699 2023-04-08 02:55:30,206 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 18c7f50b3bd243f2a74ac1ec8f70ef4e, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680922528311 2023-04-08 02:55:30,207 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 9beeb3b7e4074af9867ac03aab10856a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680922528311 2023-04-08 02:55:30,213 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 8be8843d5bbff3d4194de143513dc64b#B#compaction#414 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:30,224 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8be8843d5bbff3d4194de143513dc64b#A#compaction#413 average throughput is 1.79 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:30,269 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:30,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:30,271 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:30,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:30,272 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:30,272 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:30,272 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:30,272 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:30,272 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:30,273 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:30,273 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:30,274 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,274 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:30,274 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:30,274 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,274 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:30,274 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:30,275 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:30,276 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:30,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:30,276 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:30,276 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:30,277 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:30,277 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:30,277 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:30,277 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:30,277 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:30,277 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:30,277 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:30,278 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:30,278 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:30,278 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8be8843d5bbff3d4194de143513dc64b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-08 02:55:30,278 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=A 2023-04-08 02:55:30,278 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:30,278 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=B 2023-04-08 02:55:30,278 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:30,278 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=C 2023-04-08 02:55:30,278 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:30,634 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/00c7d74598b348debe84650d18ea1b22 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/00c7d74598b348debe84650d18ea1b22 2023-04-08 02:55:30,638 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8be8843d5bbff3d4194de143513dc64b/B of 8be8843d5bbff3d4194de143513dc64b into 00c7d74598b348debe84650d18ea1b22(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:30,638 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:30,638 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., storeName=8be8843d5bbff3d4194de143513dc64b/B, priority=12, startTime=1680922530201; duration=0sec 2023-04-08 02:55:30,638 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:30,638 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:55:30,639 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49256 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:55:30,640 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 8be8843d5bbff3d4194de143513dc64b/C is initiating minor compaction (all files) 2023-04-08 02:55:30,640 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8be8843d5bbff3d4194de143513dc64b/C in TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:30,640 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/11fc7126a9e24a288004054d40f51678, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/faed3aa5fc7d40d0aea35d921a06a751, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/1a041131b7cf419ab4d7d41cd8ef3b8c, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/be16bc1766f242fda7f1920507d21fa8] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp, totalSize=48.1 K 2023-04-08 02:55:30,640 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 11fc7126a9e24a288004054d40f51678, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=362, earliestPutTs=1680922525997 2023-04-08 02:55:30,641 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting faed3aa5fc7d40d0aea35d921a06a751, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=375, earliestPutTs=1680922526148 2023-04-08 02:55:30,641 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 1a041131b7cf419ab4d7d41cd8ef3b8c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=398, earliestPutTs=1680922526699 2023-04-08 02:55:30,641 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting be16bc1766f242fda7f1920507d21fa8, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680922528311 2023-04-08 02:55:30,648 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 8be8843d5bbff3d4194de143513dc64b#C#compaction#416 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:30,691 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408543b2b3faea94769bd2d7fb4bbafa5b5 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408543b2b3faea94769bd2d7fb4bbafa5b5 2023-04-08 02:55:30,698 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=430, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/62fd36c073c845798323acb6dc4a25cf 2023-04-08 02:55:30,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:30,848 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:30,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:30,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922590861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:30,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47734 deadline: 1680922590862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,868 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:30,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922590863, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,869 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:30,869 DEBUG [Thread-2579] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4158 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:55:30,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922590865, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,972 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:30,972 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922590968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:30,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922590970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:30,974 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:30,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922590970, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:31,048 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/117f455ed73a4d4c8755731e49e08093 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/117f455ed73a4d4c8755731e49e08093 2023-04-08 02:55:31,052 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8be8843d5bbff3d4194de143513dc64b/A of 8be8843d5bbff3d4194de143513dc64b into 117f455ed73a4d4c8755731e49e08093(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:31,052 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:31,052 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., storeName=8be8843d5bbff3d4194de143513dc64b/A, priority=12, startTime=1680922530201; duration=0sec 2023-04-08 02:55:31,052 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:31,056 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/af36662812df4f33a7025ca67fc7e370 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/af36662812df4f33a7025ca67fc7e370 2023-04-08 02:55:31,060 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 8be8843d5bbff3d4194de143513dc64b/C of 8be8843d5bbff3d4194de143513dc64b into af36662812df4f33a7025ca67fc7e370(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:31,060 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:31,060 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., storeName=8be8843d5bbff3d4194de143513dc64b/C, priority=12, startTime=1680922530201; duration=0sec 2023-04-08 02:55:31,060 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:31,114 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/7ca30188b64d4132b8b5958b5cf36a93 2023-04-08 02:55:31,176 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:31,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922591174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:31,177 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:31,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922591175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:31,177 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:31,177 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922591175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:31,191 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:31,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 391 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47700 deadline: 1680922591188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:31,480 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:31,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922591478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:31,481 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:31,481 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922591478, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:31,483 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:31,483 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922591480, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:31,523 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=430 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/4edae9673fb243d3837ebff477cc1335 2023-04-08 02:55:31,528 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/62fd36c073c845798323acb6dc4a25cf as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/62fd36c073c845798323acb6dc4a25cf 2023-04-08 02:55:31,531 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/62fd36c073c845798323acb6dc4a25cf, entries=150, sequenceid=430, filesize=25.4 K 2023-04-08 02:55:31,532 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/7ca30188b64d4132b8b5958b5cf36a93 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/7ca30188b64d4132b8b5958b5cf36a93 2023-04-08 02:55:31,535 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/7ca30188b64d4132b8b5958b5cf36a93, entries=150, sequenceid=430, filesize=11.9 K 2023-04-08 02:55:31,536 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/4edae9673fb243d3837ebff477cc1335 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/4edae9673fb243d3837ebff477cc1335 2023-04-08 02:55:31,539 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/4edae9673fb243d3837ebff477cc1335, entries=150, sequenceid=430, filesize=11.9 K 2023-04-08 02:55:31,540 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 8be8843d5bbff3d4194de143513dc64b in 1262ms, sequenceid=430, compaction requested=false 2023-04-08 02:55:31,540 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:31,540 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:31,540 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:31,540 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:31,540 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:31,540 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:31,540 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:31,541 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:31,541 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:31,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:31,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:31,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:31,542 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:31,542 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:31,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:31,542 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:31,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:31,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:31,543 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:31,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:31,543 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b1ce6a6[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:31,543 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:31,543 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:31,546 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:31,546 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:31,546 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:31,546 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:31,546 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:31,546 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:31,546 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:31,546 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:31,548 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:31,548 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:31,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:31,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:31,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:31,548 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:31,548 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:31,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:31,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:31,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:31,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:31,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:31,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:31,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:31,551 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:31,551 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:31,551 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:31,551 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:31,551 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:31,551 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:31,551 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:31,552 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:31,552 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:31,552 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:31,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:31,552 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:31,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:31,552 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:31,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:31,553 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:31,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:31,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:31,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:31,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:31,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:31,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:31,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:31,554 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:31,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:31,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:31,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:31,555 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:31,557 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:31,557 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:31,557 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:31,557 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:31,557 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:31,557 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:31,557 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:31,557 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:31,557 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:31,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:31,557 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:31,557 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:31,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:31,558 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:31,558 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:31,558 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:31,558 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:31,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:31,658 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:31,658 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:31,660 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:31,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:31,661 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:31,661 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:31,661 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:31,661 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:31,661 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:31,662 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:31,662 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:31,664 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:31,664 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:31,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:31,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:31,665 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:31,665 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:31,665 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:31,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:31,665 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:31,666 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:31,666 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:31,666 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:31,666 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:31,666 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:31,666 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:31,666 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:31,666 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:31,666 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:31,666 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8be8843d5bbff3d4194de143513dc64b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-08 02:55:31,666 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=A 2023-04-08 02:55:31,666 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:31,666 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=B 2023-04-08 02:55:31,666 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:31,666 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=C 2023-04-08 02:55:31,666 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:31,984 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:31,984 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:32,031 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:32,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922592029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:32,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:32,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922592030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:32,036 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:32,036 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922592031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:32,085 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408f137cc5721b7479ebf41b88bdad82eb4 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408f137cc5721b7479ebf41b88bdad82eb4 2023-04-08 02:55:32,135 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:32,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922592132, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:32,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:32,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922592137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:32,141 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:32,141 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922592137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:32,341 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:32,341 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 69 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922592338, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:32,343 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:32,343 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922592342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:32,347 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:32,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922592343, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:32,492 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=446, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/ddb7fac1528a4206815464709986f619 2023-04-08 02:55:32,646 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:32,646 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922592642, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:32,648 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:32,648 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922592644, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:32,652 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:32,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922592648, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:32,909 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/efad8f130bd7441d8663f92b7be06841 2023-04-08 02:55:32,918 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=446 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/5211583a4dd64d30b1cb6f7b445f573b 2023-04-08 02:55:32,923 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/ddb7fac1528a4206815464709986f619 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/ddb7fac1528a4206815464709986f619 2023-04-08 02:55:32,926 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/ddb7fac1528a4206815464709986f619, entries=150, sequenceid=446, filesize=25.4 K 2023-04-08 02:55:32,926 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/efad8f130bd7441d8663f92b7be06841 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/efad8f130bd7441d8663f92b7be06841 2023-04-08 02:55:32,931 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/efad8f130bd7441d8663f92b7be06841, entries=150, sequenceid=446, filesize=11.9 K 2023-04-08 02:55:32,932 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/5211583a4dd64d30b1cb6f7b445f573b as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/5211583a4dd64d30b1cb6f7b445f573b 2023-04-08 02:55:32,935 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/5211583a4dd64d30b1cb6f7b445f573b, entries=150, sequenceid=446, filesize=11.9 K 2023-04-08 02:55:32,936 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 8be8843d5bbff3d4194de143513dc64b in 1270ms, sequenceid=446, compaction requested=true 2023-04-08 02:55:32,936 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:32,936 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:32,936 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:32,936 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:32,936 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:32,936 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:32,936 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:32,939 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:32,939 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:32,939 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:32,940 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:32,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:32,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:32,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:32,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:32,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:32,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:32,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:32,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:32,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:32,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4892273[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:32,944 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:32,944 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:32,945 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:32,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:32,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:32,945 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:32,945 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:32,945 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:32,946 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:32,946 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:32,947 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:32,947 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:32,947 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:32,947 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:32,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:32,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:32,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:32,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:32,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:32,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:32,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:32,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:32,951 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:32,951 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:32,951 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:32,951 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:32,951 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:32,951 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:32,954 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:32,954 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:32,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:32,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:32,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:32,954 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:32,954 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:32,954 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:32,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:32,956 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:32,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:32,958 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:32,971 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:32,971 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:32,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:32,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:32,971 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:32,971 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:32,971 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:32,971 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:32,971 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:32,971 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:32,971 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:32,971 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:32,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:32,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:32,972 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:32,972 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:32,973 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:32,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:33,072 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:33,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:33,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:33,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:33,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:33,075 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:33,075 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:33,075 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:33,075 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:33,076 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:33,076 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:33,077 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:33,077 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:33,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:33,077 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,077 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:33,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:33,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:33,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:33,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:33,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:33,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:33,079 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:33,079 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:33,079 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:33,079 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:33,079 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:33,079 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:33,079 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:33,079 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:33,079 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8be8843d5bbff3d4194de143513dc64b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-08 02:55:33,079 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=A 2023-04-08 02:55:33,079 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:33,079 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=B 2023-04-08 02:55:33,080 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:33,080 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=C 2023-04-08 02:55:33,080 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:33,099 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040811250903932940c0a803c2db16d21f07 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040811250903932940c0a803c2db16d21f07 2023-04-08 02:55:33,151 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:33,151 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:33,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:33,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922593166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,172 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:33,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922593167, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,174 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:33,174 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922593170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,196 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:33,196 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 393 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47700 deadline: 1680922593194, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,198 DEBUG [Thread-2573] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4140 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:55:33,279 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:33,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922593273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,280 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:33,280 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922593273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,281 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:33,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922593274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:33,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922593482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,490 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:33,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922593482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,490 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:33,490 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 80 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922593482, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,508 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=465, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/055920780b854e1f8aaf81723e566b3e 2023-04-08 02:55:33,538 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/674601e017864935b6dab39e4a3354ba 2023-04-08 02:55:33,558 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=465 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/fdd303f5085b4952b9991b49b9f3a270 2023-04-08 02:55:33,566 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/055920780b854e1f8aaf81723e566b3e as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/055920780b854e1f8aaf81723e566b3e 2023-04-08 02:55:33,570 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/055920780b854e1f8aaf81723e566b3e, entries=150, sequenceid=465, filesize=25.4 K 2023-04-08 02:55:33,571 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/674601e017864935b6dab39e4a3354ba as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/674601e017864935b6dab39e4a3354ba 2023-04-08 02:55:33,574 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/674601e017864935b6dab39e4a3354ba, entries=150, sequenceid=465, filesize=11.9 K 2023-04-08 02:55:33,575 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/fdd303f5085b4952b9991b49b9f3a270 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/fdd303f5085b4952b9991b49b9f3a270 2023-04-08 02:55:33,579 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/fdd303f5085b4952b9991b49b9f3a270, entries=150, sequenceid=465, filesize=11.9 K 2023-04-08 02:55:33,579 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 8be8843d5bbff3d4194de143513dc64b in 500ms, sequenceid=465, compaction requested=true 2023-04-08 02:55:33,580 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:33,580 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:33,580 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:33,580 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:33,580 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,580 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:33,580 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:33,581 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:33,581 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:33,582 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:33,582 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:33,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:33,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:33,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59aded5d[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:33,583 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:33,583 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:33,585 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:33,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:33,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:33,585 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:33,585 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:33,585 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:33,586 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,586 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:33,587 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:33,587 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:33,587 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,587 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:33,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:33,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:33,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:33,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:33,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:33,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:33,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:33,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:33,590 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:33,590 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:33,590 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:33,590 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:33,590 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:33,590 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:33,592 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:33,592 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:33,592 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:33,592 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:33,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:33,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:33,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:33,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:33,593 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:33,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:33,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:33,593 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:33,595 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:33,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:33,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:33,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,602 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:33,602 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:33,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:33,603 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:33,603 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:33,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:33,603 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:33,603 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:33,603 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:33,603 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:33,603 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:33,603 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:33,603 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:33,603 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:33,703 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:33,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:33,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:33,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:33,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:33,707 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:33,707 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:33,707 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:33,707 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:33,707 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:33,707 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:33,708 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:33,708 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:33,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:33,709 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,709 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:33,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:33,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:33,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:33,710 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:33,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:33,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:33,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:33,721 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:33,721 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:33,721 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:33,721 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:33,721 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8be8843d5bbff3d4194de143513dc64b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-08 02:55:33,721 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=A 2023-04-08 02:55:33,722 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:33,722 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=B 2023-04-08 02:55:33,722 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:33,722 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=C 2023-04-08 02:55:33,722 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:33,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:33,791 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:33,830 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:33,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922593824, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:33,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922593829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,840 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:33,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922593829, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,935 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:33,935 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922593931, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:33,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922593941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:33,942 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:33,942 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922593941, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,137 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408a9d5e54c26a9497ba606e095c2a25602 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408a9d5e54c26a9497ba606e095c2a25602 2023-04-08 02:55:34,139 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:34,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922594136, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,142 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=478, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/967c18607fa045979c0c002586e12eb3 2023-04-08 02:55:34,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:34,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922594143, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,147 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:34,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922594144, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,445 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:34,446 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47724 deadline: 1680922594441, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,452 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:34,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47714 deadline: 1680922594450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:34,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922594450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,551 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/26a93f50b880417eb684cb665888fce3 2023-04-08 02:55:34,560 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=478 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/c724ad2a677d4ae1b63fbfb335ce92ea 2023-04-08 02:55:34,564 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/967c18607fa045979c0c002586e12eb3 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/967c18607fa045979c0c002586e12eb3 2023-04-08 02:55:34,568 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/967c18607fa045979c0c002586e12eb3, entries=150, sequenceid=478, filesize=25.4 K 2023-04-08 02:55:34,569 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/26a93f50b880417eb684cb665888fce3 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/26a93f50b880417eb684cb665888fce3 2023-04-08 02:55:34,573 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/26a93f50b880417eb684cb665888fce3, entries=150, sequenceid=478, filesize=11.9 K 2023-04-08 02:55:34,573 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/c724ad2a677d4ae1b63fbfb335ce92ea as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/c724ad2a677d4ae1b63fbfb335ce92ea 2023-04-08 02:55:34,577 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/c724ad2a677d4ae1b63fbfb335ce92ea, entries=150, sequenceid=478, filesize=11.9 K 2023-04-08 02:55:34,578 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 8be8843d5bbff3d4194de143513dc64b in 856ms, sequenceid=478, compaction requested=true 2023-04-08 02:55:34,578 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:34,578 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:34,578 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:34,578 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:34,578 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,578 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:34,578 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:34,580 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:34,580 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:34,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:34,580 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:34,580 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:34,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:34,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:34,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:34,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:34,581 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a74ab2[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:34,581 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:34,582 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:34,583 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:34,584 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:34,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:34,584 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:34,584 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:34,584 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:34,584 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,584 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:34,585 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:34,585 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:34,585 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,585 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:34,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:34,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:34,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:34,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:34,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:34,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:34,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:34,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:34,588 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:34,588 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:34,588 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:34,588 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:34,588 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:34,588 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:34,590 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:34,590 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:34,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:34,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:34,590 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:34,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:34,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:34,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:34,590 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:34,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:34,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:34,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:34,591 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:34,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:34,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:34,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,594 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:34,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:34,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,596 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,596 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:34,596 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:34,596 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,596 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:34,596 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:34,596 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:34,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:34,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:34,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:34,596 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:34,596 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:34,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:34,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:34,596 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:34,597 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:34,596 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:34,597 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:34,697 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:34,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:34,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:34,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:34,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:34,700 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:34,700 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:34,700 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:34,700 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:34,701 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:34,701 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:34,703 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,703 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:34,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:34,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:34,703 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,703 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:34,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:34,703 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:34,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:34,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:34,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:34,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:34,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:34,704 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:34,704 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:34,704 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:34,704 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:34,704 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:34,704 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:34,704 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:34,704 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8be8843d5bbff3d4194de143513dc64b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-08 02:55:34,705 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=A 2023-04-08 02:55:34,705 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:34,705 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=B 2023-04-08 02:55:34,705 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:34,705 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=C 2023-04-08 02:55:34,705 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:34,716 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304082bace68eaa7649ea9810350f58a25d0a to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304082bace68eaa7649ea9810350f58a25d0a 2023-04-08 02:55:34,720 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=497, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/f27f2817d6984a5da3cbb0db1cb54254 2023-04-08 02:55:34,733 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/27186e3264524586a92d4816d8021b24 2023-04-08 02:55:34,743 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=497 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/81a00314f1184d70aa63b6e5cca820dc 2023-04-08 02:55:34,746 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/f27f2817d6984a5da3cbb0db1cb54254 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/f27f2817d6984a5da3cbb0db1cb54254 2023-04-08 02:55:34,750 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/f27f2817d6984a5da3cbb0db1cb54254, entries=150, sequenceid=497, filesize=25.4 K 2023-04-08 02:55:34,751 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/27186e3264524586a92d4816d8021b24 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/27186e3264524586a92d4816d8021b24 2023-04-08 02:55:34,754 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/27186e3264524586a92d4816d8021b24, entries=150, sequenceid=497, filesize=11.9 K 2023-04-08 02:55:34,755 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/81a00314f1184d70aa63b6e5cca820dc as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/81a00314f1184d70aa63b6e5cca820dc 2023-04-08 02:55:34,759 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/81a00314f1184d70aa63b6e5cca820dc, entries=150, sequenceid=497, filesize=11.9 K 2023-04-08 02:55:34,759 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 8be8843d5bbff3d4194de143513dc64b in 55ms, sequenceid=497, compaction requested=true 2023-04-08 02:55:34,759 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:34,759 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:34,760 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:34,760 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:34,760 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,760 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:34,760 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:34,761 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:34,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:34,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:34,762 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:34,762 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:34,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:34,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:34,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:34,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:34,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:34,763 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c62b5d6[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:34,763 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:34,763 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:34,764 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:34,765 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:34,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:34,765 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:34,765 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:34,765 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:34,765 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,765 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:34,766 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:34,766 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:34,766 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:34,768 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:34,768 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:34,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:34,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:34,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:34,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:34,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:34,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:34,771 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:34,771 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:34,771 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:34,771 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:34,771 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:34,771 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:34,774 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:34,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:34,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:34,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:34,774 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:34,774 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:34,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:34,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:34,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:34,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:34,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:34,775 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:34,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:34,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:34,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:34,776 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:34,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:34,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,778 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,778 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:34,779 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:34,779 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:34,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:34,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:34,779 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:34,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:34,779 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:34,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:34,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:34,779 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:34,779 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,779 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:34,779 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:34,779 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:34,779 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:34,779 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:34,779 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:34,879 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:34,880 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:34,882 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:34,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:34,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:34,892 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:34,892 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:34,892 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:34,892 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:34,893 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:34,893 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:34,894 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,894 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:34,894 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:34,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:34,894 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,894 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:34,894 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:34,895 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:34,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:34,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:34,895 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:34,895 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:34,896 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:34,896 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:34,896 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:34,896 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:34,912 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:34,912 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:34,912 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:34,912 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:34,912 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:34,912 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:34,912 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:34,912 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:34,912 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,912 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:34,912 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:34,914 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:34,914 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:34,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:34,914 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:34,914 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:34,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:34,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:34,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:34,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:34,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:34,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21b58893[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:34,916 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:34,916 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:34,918 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:34,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:34,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:34,918 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:34,918 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:34,918 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:34,918 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,919 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:34,920 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,920 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:34,920 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:34,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:34,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:34,920 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:34,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:34,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:34,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:34,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:34,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:34,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:34,923 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:34,923 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:34,923 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:34,923 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:34,923 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:34,923 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:34,924 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:34,924 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:34,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:34,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:34,924 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:34,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:34,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:34,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:34,925 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:34,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:34,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:34,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:34,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:34,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:34,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:34,926 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:34,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:34,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,929 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:34,929 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:34,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:34,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:34,929 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:34,929 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,929 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:34,929 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:34,929 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:34,929 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:34,929 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:34,930 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:34,929 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:34,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:34,930 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:34,929 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:34,930 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:34,930 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:34,930 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:35,030 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:35,031 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:35,033 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:35,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:35,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:35,040 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:35,040 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:35,040 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:35,040 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:35,041 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:35,041 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:35,042 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:35,042 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:35,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:35,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:35,043 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:35,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:35,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:35,043 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:35,043 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:35,043 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:35,052 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:35,052 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:35,052 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:35,052 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:35,052 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:35,052 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:35,053 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:35,053 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:35,053 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8be8843d5bbff3d4194de143513dc64b 3/3 column families, dataSize=26.84 KB heapSize=80.44 KB 2023-04-08 02:55:35,054 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=A 2023-04-08 02:55:35,054 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:35,054 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=B 2023-04-08 02:55:35,054 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:35,054 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=C 2023-04-08 02:55:35,054 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:35,074 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:35,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:35,107 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304082b987056ab4a47899b20a5c24d7c2a01 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304082b987056ab4a47899b20a5c24d7c2a01 2023-04-08 02:55:35,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680922535128, max timestamp=1680922535154Num uniques 859; 2023-04-08 02:55:35,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-08 02:55:35,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680922535128, max timestamp=1680922535154Num uniques 859; 2023-04-08 02:55:35,156 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-08 02:55:35,158 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680922535128, max timestamp=1680922535154Num uniques 859; 2023-04-08 02:55:35,158 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-08 02:55:35,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680922535192, max timestamp=1680922535223Num uniques 1068; 2023-04-08 02:55:35,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680922535192, max timestamp=1680922535223Num uniques 1068; 2023-04-08 02:55:35,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-08 02:55:35,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-08 02:55:35,228 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=268, cellSize=12268, totalHeapSize=28112, min timestamp=1680922535192, max timestamp=1680922535223Num uniques 1068; 2023-04-08 02:55:35,228 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48908, new segment data size=12268, suffix heap size=139040,new segment heap size=28112 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1068, new segment cells count=268 2023-04-08 02:55:35,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680922535273, max timestamp=1680922535294Num uniques 1127; 2023-04-08 02:55:35,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680922535273, max timestamp=1680922535294Num uniques 1127; 2023-04-08 02:55:35,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-04-08 02:55:35,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=277, cellSize=12682, totalHeapSize=29048, min timestamp=1680922535273, max timestamp=1680922535294Num uniques 1127; 2023-04-08 02:55:35,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-04-08 02:55:35,297 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=51612, new segment data size=12682, suffix heap size=145176,new segment heap size=29048 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1127, new segment cells count=277 2023-04-08 02:55:35,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680922535326, max timestamp=1680922535355Num uniques 1136; 2023-04-08 02:55:35,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-08 02:55:35,357 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680922535326, max timestamp=1680922535355Num uniques 1136; 2023-04-08 02:55:35,358 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-08 02:55:35,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680922535326, max timestamp=1680922535355Num uniques 1136; 2023-04-08 02:55:35,359 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52026, new segment data size=8516, suffix heap size=146112,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1136, new segment cells count=186 2023-04-08 02:55:35,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680922535386, max timestamp=1680922535403Num uniques 1045; 2023-04-08 02:55:35,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=15800, suffix heap size=136648,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=345 2023-04-08 02:55:35,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680922535386, max timestamp=1680922535403Num uniques 1045; 2023-04-08 02:55:35,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=345, cellSize=15800, totalHeapSize=36120, min timestamp=1680922535386, max timestamp=1680922535403Num uniques 1045; 2023-04-08 02:55:35,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=15800, suffix heap size=136648,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=345 2023-04-08 02:55:35,406 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=15800, suffix heap size=136648,new segment heap size=36120 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=345 2023-04-08 02:55:35,459 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:35,459 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47704 deadline: 1680922595443, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:35,460 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680922535439, max timestamp=1680922535458Num uniques 1204; 2023-04-08 02:55:35,460 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11630, totalHeapSize=26656, min timestamp=1680922535439, max timestamp=1680922535458Num uniques 1204; 2023-04-08 02:55:35,460 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=254, cellSize=11627, totalHeapSize=26656, min timestamp=1680922535439, max timestamp=1680922535458Num uniques 1204; 2023-04-08 02:55:35,460 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=11630, suffix heap size=153184,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=254 2023-04-08 02:55:35,460 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=55140, new segment data size=11630, suffix heap size=153184,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=254 2023-04-08 02:55:35,460 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=55137, new segment data size=11627, suffix heap size=153184,new segment heap size=26656 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1204, new segment cells count=254 2023-04-08 02:55:35,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680922535495, max timestamp=1680922535506Num uniques 1113; 2023-04-08 02:55:35,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-08 02:55:35,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680922535495, max timestamp=1680922535506Num uniques 1113; 2023-04-08 02:55:35,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-08 02:55:35,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=213, cellSize=9748, totalHeapSize=22392, min timestamp=1680922535495, max timestamp=1680922535506Num uniques 1113; 2023-04-08 02:55:35,509 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50968, new segment data size=9748, suffix heap size=143720,new segment heap size=22392 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1113, new segment cells count=213 2023-04-08 02:55:35,536 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=505, memsize=8.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/0162bc3833c14848925dc9bd3adb2e78 2023-04-08 02:55:35,546 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/f8475fc7342d4b5284be3415ff2b1f8b 2023-04-08 02:55:35,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680922535561, max timestamp=1680922535576Num uniques 1072; 2023-04-08 02:55:35,578 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-08 02:55:35,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680922535561, max timestamp=1680922535576Num uniques 1072; 2023-04-08 02:55:35,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-08 02:55:35,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680922535561, max timestamp=1680922535576Num uniques 1072; 2023-04-08 02:55:35,579 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=49092, new segment data size=7872, suffix heap size=139456,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1072, new segment cells count=172 2023-04-08 02:55:35,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1680922535611, max timestamp=1680922535624Num uniques 1031; 2023-04-08 02:55:35,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=15156, suffix heap size=135192,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=331 2023-04-08 02:55:35,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1680922535611, max timestamp=1680922535624Num uniques 1031; 2023-04-08 02:55:35,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=331, cellSize=15156, totalHeapSize=34664, min timestamp=1680922535611, max timestamp=1680922535624Num uniques 1031; 2023-04-08 02:55:35,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=15156, suffix heap size=135192,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=331 2023-04-08 02:55:35,626 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=15156, suffix heap size=135192,new segment heap size=34664 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=331 2023-04-08 02:55:35,687 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:35,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47704 deadline: 1680922595676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:35,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:35,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47734 deadline: 1680922595678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:35,688 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1680922535654, max timestamp=1680922535686Num uniques 1190; 2023-04-08 02:55:35,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=54500, new segment data size=13280, suffix heap size=151728,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1190, new segment cells count=290 2023-04-08 02:55:35,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=290, cellSize=13280, totalHeapSize=30400, min timestamp=1680922535654, max timestamp=1680922535686Num uniques 1190; 2023-04-08 02:55:35,689 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=54500, new segment data size=13280, suffix heap size=151728,new segment heap size=30400 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1190, new segment cells count=290 2023-04-08 02:55:35,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680922535673, max timestamp=1680922535686Num uniques 1190; 2023-04-08 02:55:35,690 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=54500, new segment data size=10990, suffix heap size=151728,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1190, new segment cells count=240 2023-04-08 02:55:35,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680922535764, max timestamp=1680922535790Num uniques 1149; 2023-04-08 02:55:35,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-08 02:55:35,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680922535764, max timestamp=1680922535790Num uniques 1149; 2023-04-08 02:55:35,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52624, new segment data size=11404, suffix heap size=147464,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1149, new segment cells count=249 2023-04-08 02:55:35,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=249, cellSize=11404, totalHeapSize=26136, min timestamp=1680922535764, max timestamp=1680922535790Num uniques 1099; 2023-04-08 02:55:35,793 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=11404, suffix heap size=142264,new segment heap size=26136 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=249 2023-04-08 02:55:35,835 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:35,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47704 deadline: 1680922595827, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:35,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680922535814, max timestamp=1680922535834Num uniques 1108; 2023-04-08 02:55:35,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-08 02:55:35,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680922535814, max timestamp=1680922535834Num uniques 1108; 2023-04-08 02:55:35,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=208, cellSize=9520, totalHeapSize=21872, min timestamp=1680922535814, max timestamp=1680922535834Num uniques 1108; 2023-04-08 02:55:35,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-08 02:55:35,836 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50740, new segment data size=9520, suffix heap size=143200,new segment heap size=21872 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1108, new segment cells count=208 2023-04-08 02:55:35,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:35,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 157 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47724 deadline: 1680922595893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:35,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680922535882, max timestamp=1680922535901Num uniques 1067; 2023-04-08 02:55:35,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680922535882, max timestamp=1680922535901Num uniques 1067; 2023-04-08 02:55:35,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-08 02:55:35,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680922535882, max timestamp=1680922535901Num uniques 1067; 2023-04-08 02:55:35,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-08 02:55:35,903 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48862, new segment data size=7642, suffix heap size=138936,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1067, new segment cells count=167 2023-04-08 02:55:35,962 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1680922535935, max timestamp=1680922535961Num uniques 1026; 2023-04-08 02:55:35,962 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=12636, suffix heap size=134672,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=276 2023-04-08 02:55:35,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1680922535935, max timestamp=1680922535961Num uniques 1026; 2023-04-08 02:55:35,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=12636, suffix heap size=134672,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=276 2023-04-08 02:55:35,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=276, cellSize=12636, totalHeapSize=28944, min timestamp=1680922535935, max timestamp=1680922535961Num uniques 1026; 2023-04-08 02:55:35,963 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=12636, suffix heap size=134672,new segment heap size=28944 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=276 2023-04-08 02:55:35,965 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=8.95 KB at sequenceid=505 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/b24b2fb96e0b4c888129db4f06dea489 2023-04-08 02:55:35,969 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/0162bc3833c14848925dc9bd3adb2e78 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/0162bc3833c14848925dc9bd3adb2e78 2023-04-08 02:55:35,972 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/0162bc3833c14848925dc9bd3adb2e78, entries=200, sequenceid=505, filesize=32.3 K 2023-04-08 02:55:35,972 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/f8475fc7342d4b5284be3415ff2b1f8b as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/f8475fc7342d4b5284be3415ff2b1f8b 2023-04-08 02:55:35,975 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/f8475fc7342d4b5284be3415ff2b1f8b, entries=150, sequenceid=505, filesize=11.9 K 2023-04-08 02:55:35,975 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/b24b2fb96e0b4c888129db4f06dea489 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/b24b2fb96e0b4c888129db4f06dea489 2023-04-08 02:55:35,980 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/b24b2fb96e0b4c888129db4f06dea489, entries=150, sequenceid=505, filesize=11.9 K 2023-04-08 02:55:35,981 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~26.84 KB/27480, heapSize ~80.39 KB/82320, currentSize=60.38 KB/61830 for 8be8843d5bbff3d4194de143513dc64b in 928ms, sequenceid=505, compaction requested=true 2023-04-08 02:55:35,981 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:35,981 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:35,981 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:35,981 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:35,981 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:35,981 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:35,981 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:35,986 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:35,986 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:35,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:35,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:35,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:35,986 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:35,986 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:35,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:35,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:35,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:35,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:35,987 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@17c45efd[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:35,987 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:35,987 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:35,988 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:35,988 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:35,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:35,988 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:35,989 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:35,989 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:35,989 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:35,989 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:35,990 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:35,990 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:35,990 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:35,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:35,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:35,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:35,990 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:35,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:35,990 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:35,990 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8be8843d5bbff3d4194de143513dc64b 3/3 column families, dataSize=67.09 KB heapSize=174.80 KB 2023-04-08 02:55:35,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:35,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:35,991 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=A 2023-04-08 02:55:35,991 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:35,992 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=B 2023-04-08 02:55:35,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:35,992 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:35,992 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=C 2023-04-08 02:55:35,992 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:35,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:35,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:35,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:35,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:35,993 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:35,993 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:35,994 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:35,994 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:35,994 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:35,994 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:35,995 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:35,995 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:35,995 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:35,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:35,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:35,995 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:35,996 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:35,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:35,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:35,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:35,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:35,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:35,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:35,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:35,997 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:35,998 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:35,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:35,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:35,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:35,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:35,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,001 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,001 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:36,001 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:36,001 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:36,001 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:36,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:36,001 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:36,001 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,001 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:36,002 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:36,001 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:36,002 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:36,002 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,002 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:36,002 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:36,002 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:36,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:36,002 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:36,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:36,004 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304081b071b467e404d5e8431ba156665543b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304081b071b467e404d5e8431ba156665543b 2023-04-08 02:55:36,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:36,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47734 deadline: 1680922596045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,056 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:36,056 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47704 deadline: 1680922596046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:36,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47724 deadline: 1680922596056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:36,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47714 deadline: 1680922596056, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,102 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:36,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:36,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:36,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:36,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:36,105 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:36,105 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:36,106 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:36,106 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:36,106 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,106 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:36,107 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:36,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:36,107 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:36,108 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,108 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:36,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:36,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:36,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:36,108 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:36,112 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:36,112 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:36,112 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:36,112 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:36,112 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:36,115 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:36,115 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:36,115 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:36,115 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:36,115 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:36,115 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:36,115 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:36,115 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:36,115 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,115 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:36,115 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:36,117 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,117 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:36,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:36,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:36,117 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:36,117 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:36,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:36,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:36,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:36,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:36,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:36,119 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d0c746d[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:36,119 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:36,119 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:36,120 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:36,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:36,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:36,121 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:36,121 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:36,121 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:36,121 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,121 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:36,124 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:36,124 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:36,124 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:36,124 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:36,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:36,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:36,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:36,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:36,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:36,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:36,126 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:36,127 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:36,127 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:36,127 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:36,127 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:36,127 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:36,127 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:36,129 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,129 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:36,129 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,129 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:36,129 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:36,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:36,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:36,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:36,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,130 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:36,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:36,131 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:36,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,132 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:36,132 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:36,133 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,135 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:36,135 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:36,135 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:36,135 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:36,135 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,135 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:36,135 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:36,135 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,135 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:36,135 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:36,135 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:36,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:36,135 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:36,136 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:36,136 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:36,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:36,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:36,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:36,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47734 deadline: 1680922596157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,162 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:36,162 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47704 deadline: 1680922596157, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:36,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47714 deadline: 1680922596165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,172 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:36,172 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47724 deadline: 1680922596165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,236 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:36,236 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:36,238 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:36,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:36,239 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:36,239 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:36,239 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:36,239 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:36,239 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:36,239 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,239 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:36,240 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,240 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:36,241 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:36,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:36,241 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,241 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:36,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:36,241 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:36,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:36,241 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:36,242 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:36,242 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:36,242 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:36,242 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:36,242 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:36,242 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:36,242 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:36,242 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:36,242 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:36,242 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:36,242 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:36,242 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:36,242 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:36,242 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,243 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:36,243 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:36,245 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:36,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:36,245 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:36,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:36,245 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:36,245 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:36,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:36,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:36,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:36,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:36,247 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5439b81f[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:36,247 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:36,247 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:36,248 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:36,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:36,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:36,248 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:36,248 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:36,248 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:36,248 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,248 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:36,252 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,252 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:36,252 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:36,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:36,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:36,252 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:36,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:36,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:36,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:36,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:36,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:36,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,255 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:36,256 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:36,256 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:36,256 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:36,256 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:36,256 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:36,256 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:36,257 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,257 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,257 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:36,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:36,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:36,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:36,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:36,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:36,258 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:36,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:36,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:36,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,259 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:36,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:36,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,262 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,262 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:36,262 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:36,263 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:36,263 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:36,262 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:36,263 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,263 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:36,263 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:36,263 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:36,263 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:36,263 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:36,263 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:36,263 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:36,263 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:36,263 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:36,264 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:36,264 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:36,363 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:36,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:36,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:36,366 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:36,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:36,367 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:36,367 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:36,367 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:36,367 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:36,367 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,367 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:36,370 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:36,370 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:36,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:36,370 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,370 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:36,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:36,371 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:36,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:36,371 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:36,371 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:36,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:36,371 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:36,371 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:36,371 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:36,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:36,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:36,372 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:36,372 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:36,372 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:36,372 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:36,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:36,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:36,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:36,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:36,372 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:36,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 125 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:47734 deadline: 1680922596364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,374 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:36,374 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:36,374 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47704 deadline: 1680922596364, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,374 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:36,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:36,374 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:36,374 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:36,377 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:36,377 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47714 deadline: 1680922596373, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:36,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:36,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:36,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:36,384 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:36,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47724 deadline: 1680922596374, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:36,385 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59c5431b[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:36,385 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:36,385 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:36,393 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:36,393 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:36,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:36,393 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:36,393 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:36,393 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:36,394 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,394 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:36,395 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,395 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:36,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:36,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:36,395 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:36,395 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:36,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:36,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:36,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:36,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:36,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:36,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:36,398 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:36,398 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:36,398 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:36,398 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:36,398 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:36,398 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:36,401 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,401 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,401 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:36,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:36,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:36,401 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:36,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:36,401 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:36,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:36,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:36,403 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:36,403 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,404 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:36,405 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:36,406 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:36,406 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,406 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:36,407 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:36,407 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:36,407 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:36,407 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,407 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:36,407 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:36,407 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:36,407 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:36,407 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:36,407 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,407 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:36,407 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:36,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:36,407 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:36,407 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:36,408 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:36,412 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=755, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/1dc4999afbb5437eb1d8141bce247b1a 2023-04-08 02:55:36,422 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=755 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/464ad0c1e38b47ab96df41e49c751729 2023-04-08 02:55:36,435 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=755 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/98f456b65d2048b9a628d14e5e369612 2023-04-08 02:55:36,440 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/1dc4999afbb5437eb1d8141bce247b1a as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/1dc4999afbb5437eb1d8141bce247b1a 2023-04-08 02:55:36,444 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/1dc4999afbb5437eb1d8141bce247b1a, entries=250, sequenceid=755, filesize=39.2 K 2023-04-08 02:55:36,444 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/464ad0c1e38b47ab96df41e49c751729 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/464ad0c1e38b47ab96df41e49c751729 2023-04-08 02:55:36,447 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/464ad0c1e38b47ab96df41e49c751729, entries=150, sequenceid=755, filesize=11.9 K 2023-04-08 02:55:36,448 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/98f456b65d2048b9a628d14e5e369612 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/98f456b65d2048b9a628d14e5e369612 2023-04-08 02:55:36,451 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/98f456b65d2048b9a628d14e5e369612, entries=150, sequenceid=755, filesize=11.9 K 2023-04-08 02:55:36,451 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~214.59 KB/219744, currentSize=107.34 KB/109920 for 8be8843d5bbff3d4194de143513dc64b in 461ms, sequenceid=755, compaction requested=true 2023-04-08 02:55:36,451 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:36,452 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:36,452 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-08 02:55:36,452 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-08 02:55:36,452 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:36,452 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:36,455 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 230140 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-08 02:55:36,455 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 8be8843d5bbff3d4194de143513dc64b/A is initiating minor compaction (all files) 2023-04-08 02:55:36,455 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8be8843d5bbff3d4194de143513dc64b/A in TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:36,455 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 98136 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-08 02:55:36,455 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/117f455ed73a4d4c8755731e49e08093, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/62fd36c073c845798323acb6dc4a25cf, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/ddb7fac1528a4206815464709986f619, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/055920780b854e1f8aaf81723e566b3e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/967c18607fa045979c0c002586e12eb3, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/f27f2817d6984a5da3cbb0db1cb54254, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/0162bc3833c14848925dc9bd3adb2e78, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/1dc4999afbb5437eb1d8141bce247b1a] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp, totalSize=224.7 K 2023-04-08 02:55:36,455 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 8be8843d5bbff3d4194de143513dc64b/B is initiating minor compaction (all files) 2023-04-08 02:55:36,455 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8be8843d5bbff3d4194de143513dc64b/B in TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:36,455 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/00c7d74598b348debe84650d18ea1b22, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/7ca30188b64d4132b8b5958b5cf36a93, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/efad8f130bd7441d8663f92b7be06841, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/674601e017864935b6dab39e4a3354ba, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/26a93f50b880417eb684cb665888fce3, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/27186e3264524586a92d4816d8021b24, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/f8475fc7342d4b5284be3415ff2b1f8b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/464ad0c1e38b47ab96df41e49c751729] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp, totalSize=95.8 K 2023-04-08 02:55:36,456 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 117f455ed73a4d4c8755731e49e08093, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680922528311 2023-04-08 02:55:36,456 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 00c7d74598b348debe84650d18ea1b22, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680922528311 2023-04-08 02:55:36,456 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 62fd36c073c845798323acb6dc4a25cf, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680922528988 2023-04-08 02:55:36,456 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 7ca30188b64d4132b8b5958b5cf36a93, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680922528988 2023-04-08 02:55:36,456 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting ddb7fac1528a4206815464709986f619, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1680922530860 2023-04-08 02:55:36,456 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting efad8f130bd7441d8663f92b7be06841, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1680922530860 2023-04-08 02:55:36,457 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 055920780b854e1f8aaf81723e566b3e, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1680922531998 2023-04-08 02:55:36,457 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 674601e017864935b6dab39e4a3354ba, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1680922531998 2023-04-08 02:55:36,457 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 967c18607fa045979c0c002586e12eb3, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1680922533159 2023-04-08 02:55:36,457 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 26a93f50b880417eb684cb665888fce3, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1680922533159 2023-04-08 02:55:36,457 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting f27f2817d6984a5da3cbb0db1cb54254, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1680922533808 2023-04-08 02:55:36,457 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 27186e3264524586a92d4816d8021b24, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1680922533808 2023-04-08 02:55:36,457 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 0162bc3833c14848925dc9bd3adb2e78, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680922534957 2023-04-08 02:55:36,457 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting f8475fc7342d4b5284be3415ff2b1f8b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680922534957 2023-04-08 02:55:36,458 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 1dc4999afbb5437eb1d8141bce247b1a, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=755, earliestPutTs=1680922535935 2023-04-08 02:55:36,458 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 464ad0c1e38b47ab96df41e49c751729, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=755, earliestPutTs=1680922535935 2023-04-08 02:55:36,470 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 8be8843d5bbff3d4194de143513dc64b#B#compaction#438 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:36,472 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8be8843d5bbff3d4194de143513dc64b#A#compaction#437 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:36,507 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:36,508 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:36,509 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:36,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:36,511 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:36,511 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:36,511 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:36,511 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:36,511 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:36,512 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,512 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:36,515 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:36,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:36,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:36,515 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:36,515 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,515 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:36,515 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:36,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:36,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:36,516 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:36,516 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:36,516 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:36,516 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:36,516 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:36,516 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:36,516 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:36,516 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:36,516 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:36,517 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. started... 2023-04-08 02:55:36,517 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 8be8843d5bbff3d4194de143513dc64b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-08 02:55:36,517 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=A 2023-04-08 02:55:36,517 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:36,517 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=B 2023-04-08 02:55:36,517 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:36,517 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=C 2023-04-08 02:55:36,517 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:36,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:36,680 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. as already flushing 2023-04-08 02:55:36,706 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:36,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47734 deadline: 1680922596697, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,707 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:36,707 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47704 deadline: 1680922596700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:36,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47714 deadline: 1680922596700, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,710 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:36,710 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47724 deadline: 1680922596706, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:36,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47734 deadline: 1680922596807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,812 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:36,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47704 deadline: 1680922596808, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:36,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47714 deadline: 1680922596811, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:36,819 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47724 deadline: 1680922596812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:36,881 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/dd59c51b317647d891322fa7b0a0684e as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/dd59c51b317647d891322fa7b0a0684e 2023-04-08 02:55:36,886 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 8be8843d5bbff3d4194de143513dc64b/B of 8be8843d5bbff3d4194de143513dc64b into dd59c51b317647d891322fa7b0a0684e(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:36,886 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:36,886 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., storeName=8be8843d5bbff3d4194de143513dc64b/B, priority=8, startTime=1680922536452; duration=0sec 2023-04-08 02:55:36,886 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:36,886 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 8 store files, 0 compacting, 8 eligible, 16 blocking 2023-04-08 02:55:36,888 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 8 files of size 98136 starting at candidate #0 after considering 21 permutations with 21 in ratio 2023-04-08 02:55:36,889 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 8be8843d5bbff3d4194de143513dc64b/C is initiating minor compaction (all files) 2023-04-08 02:55:36,889 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8be8843d5bbff3d4194de143513dc64b/C in TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:36,889 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/af36662812df4f33a7025ca67fc7e370, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/4edae9673fb243d3837ebff477cc1335, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/5211583a4dd64d30b1cb6f7b445f573b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/fdd303f5085b4952b9991b49b9f3a270, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/c724ad2a677d4ae1b63fbfb335ce92ea, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/81a00314f1184d70aa63b6e5cca820dc, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/b24b2fb96e0b4c888129db4f06dea489, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/98f456b65d2048b9a628d14e5e369612] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp, totalSize=95.8 K 2023-04-08 02:55:36,889 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting af36662812df4f33a7025ca67fc7e370, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=411, earliestPutTs=1680922528311 2023-04-08 02:55:36,889 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 4edae9673fb243d3837ebff477cc1335, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=430, earliestPutTs=1680922528988 2023-04-08 02:55:36,890 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 5211583a4dd64d30b1cb6f7b445f573b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=446, earliestPutTs=1680922530860 2023-04-08 02:55:36,890 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting fdd303f5085b4952b9991b49b9f3a270, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=465, earliestPutTs=1680922531998 2023-04-08 02:55:36,890 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting c724ad2a677d4ae1b63fbfb335ce92ea, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=478, earliestPutTs=1680922533159 2023-04-08 02:55:36,891 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 81a00314f1184d70aa63b6e5cca820dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=497, earliestPutTs=1680922533808 2023-04-08 02:55:36,891 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting b24b2fb96e0b4c888129db4f06dea489, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=505, earliestPutTs=1680922534957 2023-04-08 02:55:36,891 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 98f456b65d2048b9a628d14e5e369612, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=755, earliestPutTs=1680922535935 2023-04-08 02:55:36,902 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 8be8843d5bbff3d4194de143513dc64b#C#compaction#440 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:36,913 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/05ffead3a8e4462c81bf14fcebf4d55f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/05ffead3a8e4462c81bf14fcebf4d55f 2023-04-08 02:55:36,917 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 8be8843d5bbff3d4194de143513dc64b/C of 8be8843d5bbff3d4194de143513dc64b into 05ffead3a8e4462c81bf14fcebf4d55f(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:36,918 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:36,918 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., storeName=8be8843d5bbff3d4194de143513dc64b/C, priority=8, startTime=1680922536452; duration=0sec 2023-04-08 02:55:36,918 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:36,941 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408aecd6e98414440d495df750cb42cfb3f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408aecd6e98414440d495df750cb42cfb3f 2023-04-08 02:55:37,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:37,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47704 deadline: 1680922597014, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:37,018 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:37,018 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47734 deadline: 1680922597015, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:37,022 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:37,022 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47714 deadline: 1680922597019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:37,024 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:37,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47724 deadline: 1680922597019, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:37,234 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:37,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 395 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47700 deadline: 1680922597226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:37,237 DEBUG [Thread-2573] client.RpcRetryingCallerImpl(129): Call exception, tries=7, retries=16, started=8179 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_0' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:55:37,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:37,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47704 deadline: 1680922597319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:37,321 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:37,321 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47734 deadline: 1680922597319, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:37,327 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:37,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47714 deadline: 1680922597324, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:37,330 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:37,330 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47724 deadline: 1680922597327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:37,340 DEBUG [Thread-2584] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x51ccd8b7 to 127.0.0.1:64826 2023-04-08 02:55:37,340 DEBUG [Thread-2584] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:55:37,343 DEBUG [Thread-2590] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d7ba242 to 127.0.0.1:64826 2023-04-08 02:55:37,343 DEBUG [Thread-2590] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:55:37,343 DEBUG [Thread-2586] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x68a0e21a to 127.0.0.1:64826 2023-04-08 02:55:37,343 DEBUG [Thread-2586] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:55:37,346 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=774, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/76684583659a40fd9715da22e67dd9db 2023-04-08 02:55:37,348 DEBUG [Thread-2588] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1ec12b02 to 127.0.0.1:64826 2023-04-08 02:55:37,348 DEBUG [Thread-2588] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:55:37,351 DEBUG [Thread-2582] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e01d5e5 to 127.0.0.1:64826 2023-04-08 02:55:37,351 DEBUG [Thread-2582] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:55:37,707 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/f663d6779de846f28a5f3570f5b8d906 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/f663d6779de846f28a5f3570f5b8d906 2023-04-08 02:55:37,710 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 8 (all) file(s) in 8be8843d5bbff3d4194de143513dc64b/A of 8be8843d5bbff3d4194de143513dc64b into f663d6779de846f28a5f3570f5b8d906(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 1sec to execute. 2023-04-08 02:55:37,710 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:37,710 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., storeName=8be8843d5bbff3d4194de143513dc64b/A, priority=8, startTime=1680922536451; duration=1sec 2023-04-08 02:55:37,710 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:37,755 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=774 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/e446e3b73ebe4e889c798a0693df72eb 2023-04-08 02:55:37,826 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:37,826 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:37,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 170 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47704 deadline: 1680922597826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:37,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47734 deadline: 1680922597826, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:37,828 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:37,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47714 deadline: 1680922597828, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:37,834 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:37,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:47724 deadline: 1680922597834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:38,163 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=774 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/e242caff382c41a084857548e32e1611 2023-04-08 02:55:38,166 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/76684583659a40fd9715da22e67dd9db as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/76684583659a40fd9715da22e67dd9db 2023-04-08 02:55:38,168 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/76684583659a40fd9715da22e67dd9db, entries=150, sequenceid=774, filesize=25.4 K 2023-04-08 02:55:38,169 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/e446e3b73ebe4e889c798a0693df72eb as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/e446e3b73ebe4e889c798a0693df72eb 2023-04-08 02:55:38,172 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/e446e3b73ebe4e889c798a0693df72eb, entries=150, sequenceid=774, filesize=11.9 K 2023-04-08 02:55:38,172 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/e242caff382c41a084857548e32e1611 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/e242caff382c41a084857548e32e1611 2023-04-08 02:55:38,175 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/e242caff382c41a084857548e32e1611, entries=150, sequenceid=774, filesize=11.9 K 2023-04-08 02:55:38,176 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 8be8843d5bbff3d4194de143513dc64b in 1659ms, sequenceid=774, compaction requested=false 2023-04-08 02:55:38,176 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:38,176 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:38,176 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:38,176 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:38,176 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:38,176 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:38,176 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:38,178 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:38,178 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:38,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:38,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:38,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:38,178 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:38,178 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:38,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:38,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:38,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:38,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:38,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:38,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:38,179 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6cc3dd68[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:38,179 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:38,179 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:38,180 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:38,180 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:38,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:38,180 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:38,180 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:38,180 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:38,180 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:38,180 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:38,183 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:38,183 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:38,183 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:38,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:38,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:38,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:38,183 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:38,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:38,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:38,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:38,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:38,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:38,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:38,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:38,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:38,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:38,185 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:38,185 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:38,185 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:38,185 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:38,185 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:38,186 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:38,186 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:38,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:38,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:38,186 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:38,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:38,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:38,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:38,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:38,186 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:38,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:38,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:38,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:38,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:38,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:38,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:38,187 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:38,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:38,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:38,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:38,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:38,189 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:38,189 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:38,189 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:38,189 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:38,189 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:38,189 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:38,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:38,189 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:38,189 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:38,189 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:38,189 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:38,189 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:38,189 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:38,189 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:38,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:38,189 DEBUG [Thread-2581] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:38,190 DEBUG [Thread-2581] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:38,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:38,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:38,290 DEBUG [Thread-2581] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:38,290 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:38,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:38,834 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 8be8843d5bbff3d4194de143513dc64b 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-08 02:55:38,834 DEBUG [Thread-2575] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x38341134 to 127.0.0.1:64826 2023-04-08 02:55:38,835 DEBUG [Thread-2575] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:55:38,835 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=A 2023-04-08 02:55:38,835 DEBUG [Thread-2579] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x3620ee35 to 127.0.0.1:64826 2023-04-08 02:55:38,835 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:38,835 DEBUG [Thread-2579] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:55:38,835 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=B 2023-04-08 02:55:38,835 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:38,835 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=C 2023-04-08 02:55:38,836 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:38,837 DEBUG [Thread-2577] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x71b5058b to 127.0.0.1:64826 2023-04-08 02:55:38,837 DEBUG [Thread-2577] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:55:38,842 DEBUG [Thread-2571] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1a5ab074 to 127.0.0.1:64826 2023-04-08 02:55:38,842 DEBUG [Thread-2571] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:55:39,257 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040824be255599524f108838e5a79d08f692 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040824be255599524f108838e5a79d08f692 2023-04-08 02:55:39,662 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=792, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/d11a6f597c814bcb81adbfca6d9cf479 2023-04-08 02:55:39,761 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-08 02:55:39,795 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/5e242e9852be4046a9669294a12af6ce, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/3f4c3bd1dc52463ca794dca0559887da, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/a9de419bf17444eb90fd56a4ef07968f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/c136c9f5586646338ceae3a588cbcf74, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/153ce75049044085b601c881407e2c2f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/998785f6ddce4510abb8deab24526ccb, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/f47e35890d97493293e04b9bafef8722, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/6564c1724fc14cd8be95892afc2e6121, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/29fcdd96a0c148959e87a373fb7599df, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/90f24717e0e5434b8d8a65a8a5c5928f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/a4d1a40f2b1a4860bba3ceef568b05e3, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/9c867c1d12b5449bb81be16cbc6d75e0, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/c3363193596a40cd814ed4c09165b120, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/709db83f505f48a69f04775a5e1b924b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/559173e8f9c9420cb1e9614c03efe583, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/582d39ea57434b568e074014f989055d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/0ee17eaa4ffd4058930b9743fca1d618, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/44bb786324104965b47c2a90b9dd854a, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/18c7f50b3bd243f2a74ac1ec8f70ef4e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/117f455ed73a4d4c8755731e49e08093, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/62fd36c073c845798323acb6dc4a25cf, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/ddb7fac1528a4206815464709986f619, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/055920780b854e1f8aaf81723e566b3e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/967c18607fa045979c0c002586e12eb3, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/f27f2817d6984a5da3cbb0db1cb54254, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/0162bc3833c14848925dc9bd3adb2e78, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/1dc4999afbb5437eb1d8141bce247b1a] to archive 2023-04-08 02:55:39,796 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 02:55:39,797 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/5e242e9852be4046a9669294a12af6ce to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/5e242e9852be4046a9669294a12af6ce 2023-04-08 02:55:39,798 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/3f4c3bd1dc52463ca794dca0559887da to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/3f4c3bd1dc52463ca794dca0559887da 2023-04-08 02:55:39,799 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/a9de419bf17444eb90fd56a4ef07968f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/a9de419bf17444eb90fd56a4ef07968f 2023-04-08 02:55:39,800 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/c136c9f5586646338ceae3a588cbcf74 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/c136c9f5586646338ceae3a588cbcf74 2023-04-08 02:55:39,801 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/153ce75049044085b601c881407e2c2f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/153ce75049044085b601c881407e2c2f 2023-04-08 02:55:39,802 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/998785f6ddce4510abb8deab24526ccb to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/998785f6ddce4510abb8deab24526ccb 2023-04-08 02:55:39,803 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/f47e35890d97493293e04b9bafef8722 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/f47e35890d97493293e04b9bafef8722 2023-04-08 02:55:39,804 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/6564c1724fc14cd8be95892afc2e6121 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/6564c1724fc14cd8be95892afc2e6121 2023-04-08 02:55:39,805 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/29fcdd96a0c148959e87a373fb7599df to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/29fcdd96a0c148959e87a373fb7599df 2023-04-08 02:55:39,806 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/90f24717e0e5434b8d8a65a8a5c5928f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/90f24717e0e5434b8d8a65a8a5c5928f 2023-04-08 02:55:39,807 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/a4d1a40f2b1a4860bba3ceef568b05e3 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/a4d1a40f2b1a4860bba3ceef568b05e3 2023-04-08 02:55:39,808 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/9c867c1d12b5449bb81be16cbc6d75e0 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/9c867c1d12b5449bb81be16cbc6d75e0 2023-04-08 02:55:39,809 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/c3363193596a40cd814ed4c09165b120 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/c3363193596a40cd814ed4c09165b120 2023-04-08 02:55:39,810 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/709db83f505f48a69f04775a5e1b924b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/709db83f505f48a69f04775a5e1b924b 2023-04-08 02:55:39,811 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/559173e8f9c9420cb1e9614c03efe583 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/559173e8f9c9420cb1e9614c03efe583 2023-04-08 02:55:39,812 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/582d39ea57434b568e074014f989055d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/582d39ea57434b568e074014f989055d 2023-04-08 02:55:39,813 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/0ee17eaa4ffd4058930b9743fca1d618 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/0ee17eaa4ffd4058930b9743fca1d618 2023-04-08 02:55:39,814 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/44bb786324104965b47c2a90b9dd854a to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/44bb786324104965b47c2a90b9dd854a 2023-04-08 02:55:39,815 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/18c7f50b3bd243f2a74ac1ec8f70ef4e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/18c7f50b3bd243f2a74ac1ec8f70ef4e 2023-04-08 02:55:39,815 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/117f455ed73a4d4c8755731e49e08093 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/117f455ed73a4d4c8755731e49e08093 2023-04-08 02:55:39,816 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/62fd36c073c845798323acb6dc4a25cf to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/62fd36c073c845798323acb6dc4a25cf 2023-04-08 02:55:39,817 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/ddb7fac1528a4206815464709986f619 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/ddb7fac1528a4206815464709986f619 2023-04-08 02:55:39,819 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/055920780b854e1f8aaf81723e566b3e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/055920780b854e1f8aaf81723e566b3e 2023-04-08 02:55:39,819 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/967c18607fa045979c0c002586e12eb3 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/967c18607fa045979c0c002586e12eb3 2023-04-08 02:55:39,820 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/f27f2817d6984a5da3cbb0db1cb54254 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/f27f2817d6984a5da3cbb0db1cb54254 2023-04-08 02:55:39,821 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/0162bc3833c14848925dc9bd3adb2e78 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/0162bc3833c14848925dc9bd3adb2e78 2023-04-08 02:55:39,822 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/1dc4999afbb5437eb1d8141bce247b1a to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/1dc4999afbb5437eb1d8141bce247b1a 2023-04-08 02:55:39,827 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/ea8bf85a01ec447d8d63ed885c143793, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/79e1becd6b73458ca1c6a2d0d88448c6, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/626491655b19477e87b536fcad936495, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/1331db03ee74421fad9569f2785d58f9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/2159c0f3754d4e1585022fd127265188, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/48a5f51d0d734396a43b78330a1dabaa, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/7c073e081ccf417182b58773a647a7b0, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/811462e7d2d348c8b5444880dfe53b8f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/13eed4c94b294bcea7b242e6541cc153, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/9f04d1f1f6824ae0ae0fcaaadb7a5b9f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/34eb14d15d0341b99b8c2ad0ab743cf1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/3a5f5d43581d41dea85b752f8a3de420, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/665807a0694a4cd49c52de79bfb59972, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/334d55cdf41b4592aab0d7cd2c55ba80, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/676c721ebef0402489a8ad65f0d6c512, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/ea64c9b0e25043088feeb44e587906e0, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/3530bf85a80d4128b435cd58e0634c82, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/ff9513a72e30469d94ec7febf405af5c, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/00c7d74598b348debe84650d18ea1b22, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/9beeb3b7e4074af9867ac03aab10856a, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/7ca30188b64d4132b8b5958b5cf36a93, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/efad8f130bd7441d8663f92b7be06841, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/674601e017864935b6dab39e4a3354ba, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/26a93f50b880417eb684cb665888fce3, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/27186e3264524586a92d4816d8021b24, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/f8475fc7342d4b5284be3415ff2b1f8b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/464ad0c1e38b47ab96df41e49c751729] to archive 2023-04-08 02:55:39,828 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 02:55:39,829 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/ea8bf85a01ec447d8d63ed885c143793 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/ea8bf85a01ec447d8d63ed885c143793 2023-04-08 02:55:39,830 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/79e1becd6b73458ca1c6a2d0d88448c6 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/79e1becd6b73458ca1c6a2d0d88448c6 2023-04-08 02:55:39,831 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/626491655b19477e87b536fcad936495 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/626491655b19477e87b536fcad936495 2023-04-08 02:55:39,831 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/1331db03ee74421fad9569f2785d58f9 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/1331db03ee74421fad9569f2785d58f9 2023-04-08 02:55:39,832 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/2159c0f3754d4e1585022fd127265188 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/2159c0f3754d4e1585022fd127265188 2023-04-08 02:55:39,833 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/48a5f51d0d734396a43b78330a1dabaa to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/48a5f51d0d734396a43b78330a1dabaa 2023-04-08 02:55:39,834 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/7c073e081ccf417182b58773a647a7b0 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/7c073e081ccf417182b58773a647a7b0 2023-04-08 02:55:39,835 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/811462e7d2d348c8b5444880dfe53b8f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/811462e7d2d348c8b5444880dfe53b8f 2023-04-08 02:55:39,836 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/13eed4c94b294bcea7b242e6541cc153 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/13eed4c94b294bcea7b242e6541cc153 2023-04-08 02:55:39,836 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/9f04d1f1f6824ae0ae0fcaaadb7a5b9f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/9f04d1f1f6824ae0ae0fcaaadb7a5b9f 2023-04-08 02:55:39,837 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/34eb14d15d0341b99b8c2ad0ab743cf1 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/34eb14d15d0341b99b8c2ad0ab743cf1 2023-04-08 02:55:39,838 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/3a5f5d43581d41dea85b752f8a3de420 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/3a5f5d43581d41dea85b752f8a3de420 2023-04-08 02:55:39,839 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/665807a0694a4cd49c52de79bfb59972 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/665807a0694a4cd49c52de79bfb59972 2023-04-08 02:55:39,840 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/334d55cdf41b4592aab0d7cd2c55ba80 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/334d55cdf41b4592aab0d7cd2c55ba80 2023-04-08 02:55:39,840 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/676c721ebef0402489a8ad65f0d6c512 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/676c721ebef0402489a8ad65f0d6c512 2023-04-08 02:55:39,841 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/ea64c9b0e25043088feeb44e587906e0 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/ea64c9b0e25043088feeb44e587906e0 2023-04-08 02:55:39,842 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/3530bf85a80d4128b435cd58e0634c82 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/3530bf85a80d4128b435cd58e0634c82 2023-04-08 02:55:39,843 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/ff9513a72e30469d94ec7febf405af5c to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/ff9513a72e30469d94ec7febf405af5c 2023-04-08 02:55:39,844 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/00c7d74598b348debe84650d18ea1b22 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/00c7d74598b348debe84650d18ea1b22 2023-04-08 02:55:39,844 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/9beeb3b7e4074af9867ac03aab10856a to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/9beeb3b7e4074af9867ac03aab10856a 2023-04-08 02:55:39,845 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/7ca30188b64d4132b8b5958b5cf36a93 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/7ca30188b64d4132b8b5958b5cf36a93 2023-04-08 02:55:39,846 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/efad8f130bd7441d8663f92b7be06841 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/efad8f130bd7441d8663f92b7be06841 2023-04-08 02:55:39,847 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/674601e017864935b6dab39e4a3354ba to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/674601e017864935b6dab39e4a3354ba 2023-04-08 02:55:39,848 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/26a93f50b880417eb684cb665888fce3 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/26a93f50b880417eb684cb665888fce3 2023-04-08 02:55:39,848 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/27186e3264524586a92d4816d8021b24 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/27186e3264524586a92d4816d8021b24 2023-04-08 02:55:39,849 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/f8475fc7342d4b5284be3415ff2b1f8b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/f8475fc7342d4b5284be3415ff2b1f8b 2023-04-08 02:55:39,850 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/464ad0c1e38b47ab96df41e49c751729 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/464ad0c1e38b47ab96df41e49c751729 2023-04-08 02:55:39,855 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] regionserver.HStore(2712): Moving the files [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/5dff424c7dce4cb488e8f94159bad612, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/9661dac69b7c4096a04d8c4b3f0e3862, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/6710cd7c3a4c438890a62cde699f457e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/edffb163eca7493981b9d94db7a53e91, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/9361748db1ed465a85392ac661f3c718, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/c1b61b2253274e889990846209926776, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/d1c075e5002c441a99d6cb840a85bda2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/32e48b7e1c0c4f2ab1eb0d8471083c18, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/48f755102cc54276b4e4ddfee5905b78, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/d39e8f81335341d984d2ee464f734285, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/916edf6e8df34fd09b14d9400b7c61cd, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/e120906470bc4bc3894269541f1c5670, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/276ce84520c1494396f2c6b363e07e26, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/4a472bf1aae8489fa078ebba0ff0f39e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/11fc7126a9e24a288004054d40f51678, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/53cba284dfd14a169a5ebb48e62d1a6b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/faed3aa5fc7d40d0aea35d921a06a751, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/1a041131b7cf419ab4d7d41cd8ef3b8c, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/af36662812df4f33a7025ca67fc7e370, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/be16bc1766f242fda7f1920507d21fa8, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/4edae9673fb243d3837ebff477cc1335, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/5211583a4dd64d30b1cb6f7b445f573b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/fdd303f5085b4952b9991b49b9f3a270, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/c724ad2a677d4ae1b63fbfb335ce92ea, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/81a00314f1184d70aa63b6e5cca820dc, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/b24b2fb96e0b4c888129db4f06dea489, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/98f456b65d2048b9a628d14e5e369612] to archive 2023-04-08 02:55:39,856 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 02:55:39,857 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/5dff424c7dce4cb488e8f94159bad612 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/5dff424c7dce4cb488e8f94159bad612 2023-04-08 02:55:39,858 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/9661dac69b7c4096a04d8c4b3f0e3862 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/9661dac69b7c4096a04d8c4b3f0e3862 2023-04-08 02:55:39,858 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/6710cd7c3a4c438890a62cde699f457e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/6710cd7c3a4c438890a62cde699f457e 2023-04-08 02:55:39,859 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/edffb163eca7493981b9d94db7a53e91 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/edffb163eca7493981b9d94db7a53e91 2023-04-08 02:55:39,860 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/9361748db1ed465a85392ac661f3c718 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/9361748db1ed465a85392ac661f3c718 2023-04-08 02:55:39,861 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/c1b61b2253274e889990846209926776 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/c1b61b2253274e889990846209926776 2023-04-08 02:55:39,861 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/d1c075e5002c441a99d6cb840a85bda2 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/d1c075e5002c441a99d6cb840a85bda2 2023-04-08 02:55:39,862 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/32e48b7e1c0c4f2ab1eb0d8471083c18 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/32e48b7e1c0c4f2ab1eb0d8471083c18 2023-04-08 02:55:39,863 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/48f755102cc54276b4e4ddfee5905b78 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/48f755102cc54276b4e4ddfee5905b78 2023-04-08 02:55:39,864 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/d39e8f81335341d984d2ee464f734285 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/d39e8f81335341d984d2ee464f734285 2023-04-08 02:55:39,864 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/916edf6e8df34fd09b14d9400b7c61cd to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/916edf6e8df34fd09b14d9400b7c61cd 2023-04-08 02:55:39,865 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/e120906470bc4bc3894269541f1c5670 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/e120906470bc4bc3894269541f1c5670 2023-04-08 02:55:39,866 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/276ce84520c1494396f2c6b363e07e26 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/276ce84520c1494396f2c6b363e07e26 2023-04-08 02:55:39,867 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/4a472bf1aae8489fa078ebba0ff0f39e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/4a472bf1aae8489fa078ebba0ff0f39e 2023-04-08 02:55:39,867 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/11fc7126a9e24a288004054d40f51678 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/11fc7126a9e24a288004054d40f51678 2023-04-08 02:55:39,868 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/53cba284dfd14a169a5ebb48e62d1a6b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/53cba284dfd14a169a5ebb48e62d1a6b 2023-04-08 02:55:39,869 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/faed3aa5fc7d40d0aea35d921a06a751 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/faed3aa5fc7d40d0aea35d921a06a751 2023-04-08 02:55:39,870 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/1a041131b7cf419ab4d7d41cd8ef3b8c to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/1a041131b7cf419ab4d7d41cd8ef3b8c 2023-04-08 02:55:39,870 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/af36662812df4f33a7025ca67fc7e370 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/af36662812df4f33a7025ca67fc7e370 2023-04-08 02:55:39,871 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/be16bc1766f242fda7f1920507d21fa8 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/be16bc1766f242fda7f1920507d21fa8 2023-04-08 02:55:39,872 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/4edae9673fb243d3837ebff477cc1335 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/4edae9673fb243d3837ebff477cc1335 2023-04-08 02:55:39,872 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/5211583a4dd64d30b1cb6f7b445f573b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/5211583a4dd64d30b1cb6f7b445f573b 2023-04-08 02:55:39,873 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/fdd303f5085b4952b9991b49b9f3a270 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/fdd303f5085b4952b9991b49b9f3a270 2023-04-08 02:55:39,874 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/c724ad2a677d4ae1b63fbfb335ce92ea to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/c724ad2a677d4ae1b63fbfb335ce92ea 2023-04-08 02:55:39,874 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/81a00314f1184d70aa63b6e5cca820dc to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/81a00314f1184d70aa63b6e5cca820dc 2023-04-08 02:55:39,875 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/b24b2fb96e0b4c888129db4f06dea489 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/b24b2fb96e0b4c888129db4f06dea489 2023-04-08 02:55:39,876 DEBUG [RS_COMPACTED_FILES_DISCHARGER-regionserver/jenkins-hbase4:0-0] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/98f456b65d2048b9a628d14e5e369612 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/98f456b65d2048b9a628d14e5e369612 2023-04-08 02:55:40,070 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=792 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/c96be4c6fa7348c1a395b5325e09551c 2023-04-08 02:55:40,082 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=792 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/3cb32e398e1a470594288c5e4cb2f55e 2023-04-08 02:55:40,089 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/d11a6f597c814bcb81adbfca6d9cf479 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/d11a6f597c814bcb81adbfca6d9cf479 2023-04-08 02:55:40,093 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/d11a6f597c814bcb81adbfca6d9cf479, entries=150, sequenceid=792, filesize=25.4 K 2023-04-08 02:55:40,094 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/c96be4c6fa7348c1a395b5325e09551c as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/c96be4c6fa7348c1a395b5325e09551c 2023-04-08 02:55:40,097 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/c96be4c6fa7348c1a395b5325e09551c, entries=150, sequenceid=792, filesize=11.9 K 2023-04-08 02:55:40,098 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/3cb32e398e1a470594288c5e4cb2f55e as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/3cb32e398e1a470594288c5e4cb2f55e 2023-04-08 02:55:40,100 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/3cb32e398e1a470594288c5e4cb2f55e, entries=150, sequenceid=792, filesize=11.9 K 2023-04-08 02:55:40,101 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=13.42 KB/13740 for 8be8843d5bbff3d4194de143513dc64b in 1267ms, sequenceid=792, compaction requested=true 2023-04-08 02:55:40,101 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:40,101 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:40,101 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:40,101 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:55:40,101 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:55:40,101 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:40,102 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 79058 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:55:40,102 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:55:40,102 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 8be8843d5bbff3d4194de143513dc64b/A is initiating minor compaction (all files) 2023-04-08 02:55:40,102 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 8be8843d5bbff3d4194de143513dc64b/B is initiating minor compaction (all files) 2023-04-08 02:55:40,102 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 8be8843d5bbff3d4194de143513dc64b/A in TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:40,102 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8be8843d5bbff3d4194de143513dc64b/B in TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:40,102 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/f663d6779de846f28a5f3570f5b8d906, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/76684583659a40fd9715da22e67dd9db, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/d11a6f597c814bcb81adbfca6d9cf479] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp, totalSize=77.2 K 2023-04-08 02:55:40,102 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/dd59c51b317647d891322fa7b0a0684e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/e446e3b73ebe4e889c798a0693df72eb, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/c96be4c6fa7348c1a395b5325e09551c] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp, totalSize=36.6 K 2023-04-08 02:55:40,103 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting f663d6779de846f28a5f3570f5b8d906, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=755, earliestPutTs=1680922535935 2023-04-08 02:55:40,103 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting dd59c51b317647d891322fa7b0a0684e, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=755, earliestPutTs=1680922535935 2023-04-08 02:55:40,103 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 76684583659a40fd9715da22e67dd9db, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=774, earliestPutTs=1680922536044 2023-04-08 02:55:40,103 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting e446e3b73ebe4e889c798a0693df72eb, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=774, earliestPutTs=1680922536044 2023-04-08 02:55:40,103 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting d11a6f597c814bcb81adbfca6d9cf479, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=792, earliestPutTs=1680922536689 2023-04-08 02:55:40,103 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting c96be4c6fa7348c1a395b5325e09551c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=792, earliestPutTs=1680922536689 2023-04-08 02:55:40,113 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 8be8843d5bbff3d4194de143513dc64b#B#compaction#446 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:40,125 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 8be8843d5bbff3d4194de143513dc64b#A#compaction#447 average throughput is 1.97 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:40,127 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/57e3fe2bf34d4acca1554a8723e0d3b0 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/57e3fe2bf34d4acca1554a8723e0d3b0 2023-04-08 02:55:40,131 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8be8843d5bbff3d4194de143513dc64b/B of 8be8843d5bbff3d4194de143513dc64b into 57e3fe2bf34d4acca1554a8723e0d3b0(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:40,131 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:40,131 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., storeName=8be8843d5bbff3d4194de143513dc64b/B, priority=13, startTime=1680922540101; duration=0sec 2023-04-08 02:55:40,131 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:40,131 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:55:40,132 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37478 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:55:40,132 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 8be8843d5bbff3d4194de143513dc64b/C is initiating minor compaction (all files) 2023-04-08 02:55:40,132 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 8be8843d5bbff3d4194de143513dc64b/C in TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:40,133 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/05ffead3a8e4462c81bf14fcebf4d55f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/e242caff382c41a084857548e32e1611, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/3cb32e398e1a470594288c5e4cb2f55e] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp, totalSize=36.6 K 2023-04-08 02:55:40,133 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 05ffead3a8e4462c81bf14fcebf4d55f, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=755, earliestPutTs=1680922535935 2023-04-08 02:55:40,133 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting e242caff382c41a084857548e32e1611, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=774, earliestPutTs=1680922536044 2023-04-08 02:55:40,134 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 3cb32e398e1a470594288c5e4cb2f55e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=792, earliestPutTs=1680922536689 2023-04-08 02:55:40,151 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 8be8843d5bbff3d4194de143513dc64b#C#compaction#448 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:40,541 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/c6cf032d2b4e4825a76f3073874d799f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/c6cf032d2b4e4825a76f3073874d799f 2023-04-08 02:55:40,545 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8be8843d5bbff3d4194de143513dc64b/A of 8be8843d5bbff3d4194de143513dc64b into c6cf032d2b4e4825a76f3073874d799f(size=25.5 K), total size for store is 25.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:40,545 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:40,545 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., storeName=8be8843d5bbff3d4194de143513dc64b/A, priority=13, startTime=1680922540101; duration=0sec 2023-04-08 02:55:40,546 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:40,560 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/7d56861852c64ee78754d5252792478a as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/7d56861852c64ee78754d5252792478a 2023-04-08 02:55:40,563 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 8be8843d5bbff3d4194de143513dc64b/C of 8be8843d5bbff3d4194de143513dc64b into 7d56861852c64ee78754d5252792478a(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:40,563 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:40,563 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b., storeName=8be8843d5bbff3d4194de143513dc64b/C, priority=13, startTime=1680922540101; duration=0sec 2023-04-08 02:55:40,564 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:47,320 DEBUG [Thread-2573] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0bcfc1e0 to 127.0.0.1:64826 2023-04-08 02:55:47,320 DEBUG [Thread-2573] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:55:47,320 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-08 02:55:47,320 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 97 2023-04-08 02:55:47,320 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 318 2023-04-08 02:55:47,320 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 85 2023-04-08 02:55:47,321 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 112 2023-04-08 02:55:47,321 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 85 2023-04-08 02:55:47,321 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-08 02:55:47,321 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-08 02:55:47,321 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(402): scanned 1999 2023-04-08 02:55:47,321 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(403): verified 5996 rows 2023-04-08 02:55:47,321 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(402): scanned 1994 2023-04-08 02:55:47,321 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(403): verified 5978 rows 2023-04-08 02:55:47,321 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(402): scanned 2015 2023-04-08 02:55:47,321 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(403): verified 6044 rows 2023-04-08 02:55:47,321 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(402): scanned 1998 2023-04-08 02:55:47,321 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(403): verified 5993 rows 2023-04-08 02:55:47,321 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(402): scanned 2019 2023-04-08 02:55:47,321 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(403): verified 6056 rows 2023-04-08 02:55:47,321 INFO [Listener at localhost/39741] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-08 02:55:47,321 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x0fd3d54e to 127.0.0.1:64826 2023-04-08 02:55:47,321 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:55:47,323 INFO [Listener at localhost/39741] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-08 02:55:47,323 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-08 02:55:47,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure2.ProcedureExecutor(1029): Stored pid=43, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-08 02:55:47,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-08 02:55:47,328 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680922547327"}]},"ts":"1680922547327"} 2023-04-08 02:55:47,328 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-08 02:55:47,331 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-08 02:55:47,331 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8be8843d5bbff3d4194de143513dc64b, UNASSIGN}] 2023-04-08 02:55:47,333 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=44, ppid=43, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=8be8843d5bbff3d4194de143513dc64b, UNASSIGN 2023-04-08 02:55:47,333 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=8be8843d5bbff3d4194de143513dc64b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:47,333 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922547333"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680922547333"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680922547333"}]},"ts":"1680922547333"} 2023-04-08 02:55:47,334 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=45, ppid=44, state=RUNNABLE; CloseRegionProcedure 8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983}] 2023-04-08 02:55:47,428 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-08 02:55:47,486 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:47,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 8be8843d5bbff3d4194de143513dc64b, disabling compactions & flushes 2023-04-08 02:55:47,486 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:47,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:47,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. after waiting 0 ms 2023-04-08 02:55:47,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:47,486 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 8be8843d5bbff3d4194de143513dc64b 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-08 02:55:47,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=A 2023-04-08 02:55:47,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:47,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=B 2023-04-08 02:55:47,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:47,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 8be8843d5bbff3d4194de143513dc64b, store=C 2023-04-08 02:55:47,486 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:47,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-08 02:55:47,896 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408398ea288b2244cc1a59a9f32e264ed1e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408398ea288b2244cc1a59a9f32e264ed1e 2023-04-08 02:55:47,929 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-08 02:55:48,300 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=801, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/8a360ec992a643d48df9a4b11db716b7 2023-04-08 02:55:48,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-08 02:55:48,708 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=801 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/27af0e9be6da42ce994508357f8cc77f 2023-04-08 02:55:49,115 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=801 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/3fcd053196cd4badaa101bd907246ab0 2023-04-08 02:55:49,119 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/A/8a360ec992a643d48df9a4b11db716b7 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/8a360ec992a643d48df9a4b11db716b7 2023-04-08 02:55:49,122 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/8a360ec992a643d48df9a4b11db716b7, entries=100, sequenceid=801, filesize=18.6 K 2023-04-08 02:55:49,123 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/B/27af0e9be6da42ce994508357f8cc77f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/27af0e9be6da42ce994508357f8cc77f 2023-04-08 02:55:49,127 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/27af0e9be6da42ce994508357f8cc77f, entries=100, sequenceid=801, filesize=9.5 K 2023-04-08 02:55:49,127 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/.tmp/C/3fcd053196cd4badaa101bd907246ab0 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/3fcd053196cd4badaa101bd907246ab0 2023-04-08 02:55:49,131 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/3fcd053196cd4badaa101bd907246ab0, entries=100, sequenceid=801, filesize=9.5 K 2023-04-08 02:55:49,131 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 8be8843d5bbff3d4194de143513dc64b in 1645ms, sequenceid=801, compaction requested=false 2023-04-08 02:55:49,134 DEBUG [StoreCloser-TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/f663d6779de846f28a5f3570f5b8d906, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/76684583659a40fd9715da22e67dd9db, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/d11a6f597c814bcb81adbfca6d9cf479] to archive 2023-04-08 02:55:49,135 DEBUG [StoreCloser-TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 02:55:49,136 DEBUG [StoreCloser-TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/f663d6779de846f28a5f3570f5b8d906 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/f663d6779de846f28a5f3570f5b8d906 2023-04-08 02:55:49,137 DEBUG [StoreCloser-TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/76684583659a40fd9715da22e67dd9db to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/76684583659a40fd9715da22e67dd9db 2023-04-08 02:55:49,138 DEBUG [StoreCloser-TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/d11a6f597c814bcb81adbfca6d9cf479 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/d11a6f597c814bcb81adbfca6d9cf479 2023-04-08 02:55:49,143 DEBUG [StoreCloser-TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/dd59c51b317647d891322fa7b0a0684e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/e446e3b73ebe4e889c798a0693df72eb, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/c96be4c6fa7348c1a395b5325e09551c] to archive 2023-04-08 02:55:49,143 DEBUG [StoreCloser-TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 02:55:49,145 DEBUG [StoreCloser-TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/dd59c51b317647d891322fa7b0a0684e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/dd59c51b317647d891322fa7b0a0684e 2023-04-08 02:55:49,146 DEBUG [StoreCloser-TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/e446e3b73ebe4e889c798a0693df72eb to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/e446e3b73ebe4e889c798a0693df72eb 2023-04-08 02:55:49,147 DEBUG [StoreCloser-TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/c96be4c6fa7348c1a395b5325e09551c to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/c96be4c6fa7348c1a395b5325e09551c 2023-04-08 02:55:49,151 DEBUG [StoreCloser-TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/05ffead3a8e4462c81bf14fcebf4d55f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/e242caff382c41a084857548e32e1611, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/3cb32e398e1a470594288c5e4cb2f55e] to archive 2023-04-08 02:55:49,151 DEBUG [StoreCloser-TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 02:55:49,153 DEBUG [StoreCloser-TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/05ffead3a8e4462c81bf14fcebf4d55f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/05ffead3a8e4462c81bf14fcebf4d55f 2023-04-08 02:55:49,154 DEBUG [StoreCloser-TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/e242caff382c41a084857548e32e1611 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/e242caff382c41a084857548e32e1611 2023-04-08 02:55:49,155 DEBUG [StoreCloser-TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/3cb32e398e1a470594288c5e4cb2f55e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/3cb32e398e1a470594288c5e4cb2f55e 2023-04-08 02:55:49,163 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/recovered.edits/804.seqid, newMaxSeqId=804, maxSeqId=4 2023-04-08 02:55:49,164 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b. 2023-04-08 02:55:49,164 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 8be8843d5bbff3d4194de143513dc64b: 2023-04-08 02:55:49,165 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:49,165 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=44 updating hbase:meta row=8be8843d5bbff3d4194de143513dc64b, regionState=CLOSED 2023-04-08 02:55:49,166 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922549165"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680922549165"}]},"ts":"1680922549165"} 2023-04-08 02:55:49,168 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=45, resume processing ppid=44 2023-04-08 02:55:49,168 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=45, ppid=44, state=SUCCESS; CloseRegionProcedure 8be8843d5bbff3d4194de143513dc64b, server=jenkins-hbase4.apache.org,35089,1680922417983 in 1.8320 sec 2023-04-08 02:55:49,169 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=44, resume processing ppid=43 2023-04-08 02:55:49,169 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=44, ppid=43, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=8be8843d5bbff3d4194de143513dc64b, UNASSIGN in 1.8370 sec 2023-04-08 02:55:49,170 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680922549169"}]},"ts":"1680922549169"} 2023-04-08 02:55:49,170 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-08 02:55:49,172 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-08 02:55:49,174 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=43, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.8500 sec 2023-04-08 02:55:49,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=43 2023-04-08 02:55:49,431 INFO [Listener at localhost/39741] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 43 completed 2023-04-08 02:55:49,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-08 02:55:49,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure2.ProcedureExecutor(1029): Stored pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:55:49,434 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=46, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:55:49,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-08 02:55:49,434 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=46, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:55:49,440 DEBUG [HFileArchiver-9] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:49,441 DEBUG [HFileArchiver-9] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A, FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B, FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C, FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/recovered.edits] 2023-04-08 02:55:49,444 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/8a360ec992a643d48df9a4b11db716b7 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/8a360ec992a643d48df9a4b11db716b7 2023-04-08 02:55:49,445 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/c6cf032d2b4e4825a76f3073874d799f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/A/c6cf032d2b4e4825a76f3073874d799f 2023-04-08 02:55:49,447 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/27af0e9be6da42ce994508357f8cc77f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/27af0e9be6da42ce994508357f8cc77f 2023-04-08 02:55:49,448 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/57e3fe2bf34d4acca1554a8723e0d3b0 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/B/57e3fe2bf34d4acca1554a8723e0d3b0 2023-04-08 02:55:49,450 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/3fcd053196cd4badaa101bd907246ab0 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/3fcd053196cd4badaa101bd907246ab0 2023-04-08 02:55:49,451 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/7d56861852c64ee78754d5252792478a to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/C/7d56861852c64ee78754d5252792478a 2023-04-08 02:55:49,453 DEBUG [HFileArchiver-9] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/recovered.edits/804.seqid to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b/recovered.edits/804.seqid 2023-04-08 02:55:49,454 DEBUG [HFileArchiver-9] backup.HFileArchiver(596): Deleted hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/8be8843d5bbff3d4194de143513dc64b 2023-04-08 02:55:49,454 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-08 02:55:49,454 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-08 02:55:49,455 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-08 02:55:49,457 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040811250903932940c0a803c2db16d21f07 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040811250903932940c0a803c2db16d21f07 2023-04-08 02:55:49,458 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304081b071b467e404d5e8431ba156665543b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304081b071b467e404d5e8431ba156665543b 2023-04-08 02:55:49,460 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408214d79f4e1f64ae79b227c679999d7c4 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408214d79f4e1f64ae79b227c679999d7c4 2023-04-08 02:55:49,461 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040824be255599524f108838e5a79d08f692 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040824be255599524f108838e5a79d08f692 2023-04-08 02:55:49,462 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304082b987056ab4a47899b20a5c24d7c2a01 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304082b987056ab4a47899b20a5c24d7c2a01 2023-04-08 02:55:49,463 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304082bace68eaa7649ea9810350f58a25d0a to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304082bace68eaa7649ea9810350f58a25d0a 2023-04-08 02:55:49,464 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304082ead7753fecf4acf812f1e20fb0a171c to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304082ead7753fecf4acf812f1e20fb0a171c 2023-04-08 02:55:49,465 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304083296659b77b046148e5e85b591515139 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304083296659b77b046148e5e85b591515139 2023-04-08 02:55:49,466 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408398ea288b2244cc1a59a9f32e264ed1e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408398ea288b2244cc1a59a9f32e264ed1e 2023-04-08 02:55:49,467 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304083ada5ac2e61449e88a2b66cb905b7317 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304083ada5ac2e61449e88a2b66cb905b7317 2023-04-08 02:55:49,468 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408543b2b3faea94769bd2d7fb4bbafa5b5 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408543b2b3faea94769bd2d7fb4bbafa5b5 2023-04-08 02:55:49,469 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304085839627db72845f78c1517e08a5b06a8 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304085839627db72845f78c1517e08a5b06a8 2023-04-08 02:55:49,473 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040869a300c1c1114b9cb141a7f834d2978e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040869a300c1c1114b9cb141a7f834d2978e 2023-04-08 02:55:49,476 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040870560bd0651949dfbd3ca5eb9cfc94d1 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040870560bd0651949dfbd3ca5eb9cfc94d1 2023-04-08 02:55:49,477 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040894138417d1994606947eb94b4e3a8444 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040894138417d1994606947eb94b4e3a8444 2023-04-08 02:55:49,478 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408a2739c9528454ede98a08ae3f0e166b6 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408a2739c9528454ede98a08ae3f0e166b6 2023-04-08 02:55:49,479 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408a9d5e54c26a9497ba606e095c2a25602 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408a9d5e54c26a9497ba606e095c2a25602 2023-04-08 02:55:49,481 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408a9e3b9a2e76641678c5ff09c4c4e6efb to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408a9e3b9a2e76641678c5ff09c4c4e6efb 2023-04-08 02:55:49,482 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408aecd6e98414440d495df750cb42cfb3f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408aecd6e98414440d495df750cb42cfb3f 2023-04-08 02:55:49,483 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408ce0507f8586a459bbcfbe2ef48f3a779 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408ce0507f8586a459bbcfbe2ef48f3a779 2023-04-08 02:55:49,484 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408d246150270d144b9a1fcfbe45f36ca36 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408d246150270d144b9a1fcfbe45f36ca36 2023-04-08 02:55:49,485 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408f137cc5721b7479ebf41b88bdad82eb4 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408f137cc5721b7479ebf41b88bdad82eb4 2023-04-08 02:55:49,486 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408fa8a5a97150246e48038011d5bea0d46 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408fa8a5a97150246e48038011d5bea0d46 2023-04-08 02:55:49,487 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408fc0e371d74784157ace9f2ba8d499dd9 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408fc0e371d74784157ace9f2ba8d499dd9 2023-04-08 02:55:49,488 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408fecabb72cfb844cabc7503fc080ab620 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408fecabb72cfb844cabc7503fc080ab620 2023-04-08 02:55:49,489 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-08 02:55:49,490 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=46, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:55:49,492 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-08 02:55:49,493 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-08 02:55:49,494 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=46, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:55:49,494 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-08 02:55:49,494 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680922549494"}]},"ts":"9223372036854775807"} 2023-04-08 02:55:49,495 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-08 02:55:49,495 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 8be8843d5bbff3d4194de143513dc64b, NAME => 'TestAcidGuarantees,,1680922514259.8be8843d5bbff3d4194de143513dc64b.', STARTKEY => '', ENDKEY => ''}] 2023-04-08 02:55:49,495 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-08 02:55:49,496 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680922549495"}]},"ts":"9223372036854775807"} 2023-04-08 02:55:49,497 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-08 02:55:49,499 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=46, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:55:49,499 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=46, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 67 msec 2023-04-08 02:55:49,535 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=46 2023-04-08 02:55:49,535 INFO [Listener at localhost/39741] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 46 completed 2023-04-08 02:55:49,549 INFO [Listener at localhost/39741] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobScanAtomicity Thread=227 (was 225) - Thread LEAK? -, OpenFileDescriptor=517 (was 579), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=316 (was 401), ProcessCount=172 (was 170) - ProcessCount LEAK? -, AvailableMemoryMB=1372 (was 1465) 2023-04-08 02:55:49,561 INFO [Listener at localhost/39741] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=227, OpenFileDescriptor=517, MaxFileDescriptor=60000, SystemLoadAverage=316, ProcessCount=172, AvailableMemoryMB=1372 2023-04-08 02:55:49,563 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-08 02:55:49,564 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-08 02:55:49,565 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure2.ProcedureExecutor(1029): Stored pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-08 02:55:49,566 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-08 02:55:49,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 47 2023-04-08 02:55:49,567 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-08 02:55:49,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-08 02:55:49,567 DEBUG [HFileArchiver-10] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:55:49,568 DEBUG [HFileArchiver-10] backup.HFileArchiver(153): Directory hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767 empty. 2023-04-08 02:55:49,568 DEBUG [HFileArchiver-10] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:55:49,568 DEBUG [PEWorker-4] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-08 02:55:49,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-08 02:55:49,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-08 02:55:49,979 DEBUG [PEWorker-4] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-08 02:55:49,980 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 61e4c342830c067a5389bf73aa5cd767, NAME => 'TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp 2023-04-08 02:55:50,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-08 02:55:50,390 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 02:55:50,390 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 61e4c342830c067a5389bf73aa5cd767, disabling compactions & flushes 2023-04-08 02:55:50,390 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:50,390 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:50,390 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. after waiting 0 ms 2023-04-08 02:55:50,390 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:50,390 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:50,390 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:55:50,392 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-08 02:55:50,393 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922550393"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680922550393"}]},"ts":"1680922550393"} 2023-04-08 02:55:50,394 INFO [PEWorker-4] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-08 02:55:50,394 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-08 02:55:50,395 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680922550395"}]},"ts":"1680922550395"} 2023-04-08 02:55:50,395 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-08 02:55:50,398 INFO [PEWorker-4] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=61e4c342830c067a5389bf73aa5cd767, ASSIGN}] 2023-04-08 02:55:50,399 INFO [PEWorker-2] procedure.MasterProcedureScheduler(727): Took xlock for pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=61e4c342830c067a5389bf73aa5cd767, ASSIGN 2023-04-08 02:55:50,400 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=48, ppid=47, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=61e4c342830c067a5389bf73aa5cd767, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35089,1680922417983; forceNewPlan=false, retain=false 2023-04-08 02:55:50,551 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=61e4c342830c067a5389bf73aa5cd767, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:50,551 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922550551"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680922550551"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680922550551"}]},"ts":"1680922550551"} 2023-04-08 02:55:50,552 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=49, ppid=48, state=RUNNABLE; OpenRegionProcedure 61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983}] 2023-04-08 02:55:50,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-08 02:55:50,707 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:50,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 61e4c342830c067a5389bf73aa5cd767, NAME => 'TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.', STARTKEY => '', ENDKEY => ''} 2023-04-08 02:55:50,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:55:50,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 02:55:50,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:55:50,707 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:55:50,708 INFO [StoreOpener-61e4c342830c067a5389bf73aa5cd767-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:55:50,710 DEBUG [StoreOpener-61e4c342830c067a5389bf73aa5cd767-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A 2023-04-08 02:55:50,710 DEBUG [StoreOpener-61e4c342830c067a5389bf73aa5cd767-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A 2023-04-08 02:55:50,710 INFO [StoreOpener-61e4c342830c067a5389bf73aa5cd767-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 02:55:50,710 INFO [StoreOpener-61e4c342830c067a5389bf73aa5cd767-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 61e4c342830c067a5389bf73aa5cd767 columnFamilyName A 2023-04-08 02:55:50,711 INFO [StoreOpener-61e4c342830c067a5389bf73aa5cd767-1] regionserver.HStore(310): Store=61e4c342830c067a5389bf73aa5cd767/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:55:50,711 INFO [StoreOpener-61e4c342830c067a5389bf73aa5cd767-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:55:50,712 DEBUG [StoreOpener-61e4c342830c067a5389bf73aa5cd767-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B 2023-04-08 02:55:50,712 DEBUG [StoreOpener-61e4c342830c067a5389bf73aa5cd767-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B 2023-04-08 02:55:50,712 INFO [StoreOpener-61e4c342830c067a5389bf73aa5cd767-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 02:55:50,712 INFO [StoreOpener-61e4c342830c067a5389bf73aa5cd767-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 61e4c342830c067a5389bf73aa5cd767 columnFamilyName B 2023-04-08 02:55:50,712 INFO [StoreOpener-61e4c342830c067a5389bf73aa5cd767-1] regionserver.HStore(310): Store=61e4c342830c067a5389bf73aa5cd767/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:55:50,712 INFO [StoreOpener-61e4c342830c067a5389bf73aa5cd767-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:55:50,713 DEBUG [StoreOpener-61e4c342830c067a5389bf73aa5cd767-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C 2023-04-08 02:55:50,713 DEBUG [StoreOpener-61e4c342830c067a5389bf73aa5cd767-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C 2023-04-08 02:55:50,713 INFO [StoreOpener-61e4c342830c067a5389bf73aa5cd767-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 02:55:50,714 INFO [StoreOpener-61e4c342830c067a5389bf73aa5cd767-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 61e4c342830c067a5389bf73aa5cd767 columnFamilyName C 2023-04-08 02:55:50,714 INFO [StoreOpener-61e4c342830c067a5389bf73aa5cd767-1] regionserver.HStore(310): Store=61e4c342830c067a5389bf73aa5cd767/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:55:50,714 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:50,715 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:55:50,715 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:55:50,717 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-08 02:55:50,717 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:55:50,718 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-08 02:55:50,719 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 61e4c342830c067a5389bf73aa5cd767; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=73958671, jitterRate=0.10207007825374603}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-08 02:55:50,719 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:55:50,719 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., pid=49, masterSystemTime=1680922550704 2023-04-08 02:55:50,721 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:50,721 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:50,721 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=48 updating hbase:meta row=61e4c342830c067a5389bf73aa5cd767, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:50,721 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922550721"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680922550721"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680922550721"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680922550721"}]},"ts":"1680922550721"} 2023-04-08 02:55:50,723 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=49, resume processing ppid=48 2023-04-08 02:55:50,723 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=49, ppid=48, state=SUCCESS; OpenRegionProcedure 61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 in 170 msec 2023-04-08 02:55:50,724 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=48, resume processing ppid=47 2023-04-08 02:55:50,724 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=48, ppid=47, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=61e4c342830c067a5389bf73aa5cd767, ASSIGN in 325 msec 2023-04-08 02:55:50,725 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-08 02:55:50,725 DEBUG [PEWorker-2] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680922550725"}]},"ts":"1680922550725"} 2023-04-08 02:55:50,726 INFO [PEWorker-2] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-08 02:55:50,729 INFO [PEWorker-2] procedure.CreateTableProcedure(80): pid=47, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-08 02:55:50,730 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=47, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1650 sec 2023-04-08 02:55:51,671 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=47 2023-04-08 02:55:51,671 INFO [Listener at localhost/39741] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 47 completed 2023-04-08 02:55:51,672 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x5e372785 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:55:51,677 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2a987907, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:55:51,678 DEBUG [Listener at localhost/39741] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:55:51,680 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50174, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:55:51,681 DEBUG [Listener at localhost/39741] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-08 02:55:51,683 INFO [RS-EventLoopGroup-1-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:33726, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-08 02:55:51,684 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x501c5a88 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:55:51,688 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@37705401, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:55:51,689 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x27f2d997 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:55:51,693 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@15880ffb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:55:51,693 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x34ad4aa8 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:55:51,698 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@52203cb, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:55:51,698 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x1c29757d to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:55:51,702 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@420f945a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:55:51,702 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x06edd017 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:55:51,706 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@51b8df2a, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:55:51,706 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x67346c47 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:55:51,710 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@70e2be04, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:55:51,710 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x426cb82a to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:55:51,717 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@67ea274, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:55:51,717 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x5f5363d8 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:55:51,723 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@78592df0, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:55:51,724 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x211582f2 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:55:51,728 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@659c68ab, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:55:51,728 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x65816b6e to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:55:51,738 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@680e773e, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:55:51,744 DEBUG [hconnection-0x7a231dea-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:55:51,749 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50180, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:55:51,750 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:51,758 DEBUG [hconnection-0xa5a7a8f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:55:51,758 DEBUG [hconnection-0x17649fcb-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:55:51,758 DEBUG [hconnection-0x46305e9b-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:55:51,759 DEBUG [hconnection-0x680bcd86-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:55:51,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:51,760 DEBUG [hconnection-0x61511f3e-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:55:51,760 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50188, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:55:51,760 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50190, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:55:51,762 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:51,762 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:51,762 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:51,762 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:51,762 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:51,763 DEBUG [hconnection-0x14f09f19-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:55:51,765 DEBUG [hconnection-0xe088143-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:55:51,766 DEBUG [hconnection-0x4acea22f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:55:51,767 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50216, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:55:51,768 DEBUG [hconnection-0x4ac85646-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:55:51,768 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50232, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:55:51,768 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50200, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:55:51,769 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:51,771 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:51,771 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50238, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:55:51,776 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:51,776 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:51,777 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50260, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:55:51,778 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50250, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:55:51,778 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,778 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:51,778 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:50252, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:55:51,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:51,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:51,784 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:51,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:51,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:51,785 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:51,789 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:51,790 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:51,790 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:55:51,791 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-08 02:55:51,791 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:51,791 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:51,791 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:51,791 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:51,791 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:51,791 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:51,791 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:51,791 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:51,791 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:51,791 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:51,791 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:51,791 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,792 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:51,792 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:51,793 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:55:51,793 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:51,793 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:55:51,793 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:51,794 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:55:51,794 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:51,797 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,797 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:51,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:51,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:51,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:51,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:51,798 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:51,798 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:51,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:51,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:51,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:51,799 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22227381[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:51,799 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:51,799 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:51,801 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:51,801 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:51,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:51,801 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:51,801 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:51,801 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:51,801 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,801 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:51,808 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:51,808 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:51,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,808 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:51,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:51,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:51,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:51,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:51,810 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922611805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,810 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:51,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922611807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:51,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:51,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,811 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:51,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922611810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:51,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:51,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922611810, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,812 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:51,812 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:51,812 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:51,812 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:51,812 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:51,812 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:51,812 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:51,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 8 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922611812, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,814 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:51,814 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:51,814 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:51,814 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:51,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:51,815 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,815 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:51,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:51,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:51,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,817 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:51,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,819 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:51,819 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,819 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:51,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:51,819 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:51,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:51,819 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:51,819 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:51,819 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:51,819 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:51,819 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:51,819 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,819 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:51,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:51,819 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:51,819 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:51,820 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:51,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:51,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922611912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,914 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:51,914 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922611912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,916 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:51,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922611913, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,919 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:51,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:51,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:51,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922611917, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,920 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:51,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922611918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:51,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:51,925 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:51,925 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:51,926 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:51,926 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:51,926 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:51,926 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:51,926 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:51,927 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:51,927 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,927 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:51,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:51,928 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,928 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:51,928 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:51,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:51,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:51,929 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:51,929 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:51,929 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:51,929 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:51,929 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:51,929 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:51,929 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:51,929 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:51,929 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:51,929 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:51,929 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:51,929 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:51,930 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:51,930 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:51,930 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,930 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:51,930 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:51,936 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:51,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:51,937 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:51,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:51,939 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:51,940 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3bc2de9a[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:51,940 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:51,940 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:51,941 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:51,941 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:51,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:51,941 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:51,941 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,941 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:51,941 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:51,941 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:51,942 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,942 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:51,942 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:51,942 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:51,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:51,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:51,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:51,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:51,945 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:51,945 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:51,945 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:51,945 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:51,945 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:51,945 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:51,947 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:51,947 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:51,947 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:51,947 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:51,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:51,947 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,948 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:51,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:51,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:51,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:51,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:51,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,950 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:51,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:51,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,954 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,954 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:51,954 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:51,954 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:51,954 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:51,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:51,954 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:51,954 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:51,954 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:51,954 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:51,954 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:51,954 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:51,954 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:51,954 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:51,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:51,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:51,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:51,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:52,054 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:52,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:52,057 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:52,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:52,058 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:52,058 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:52,058 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:52,058 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:52,058 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:52,058 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:52,058 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:52,059 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:52,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:52,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:52,060 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:52,060 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,060 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:52,061 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:52,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:52,061 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:52,062 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:52,062 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:52,062 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:52,062 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:52,062 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:52,062 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:52,062 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:52,062 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:52,062 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:52,062 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:52,063 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:52,063 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:52,063 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:52,063 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,063 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:52,063 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:52,064 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:52,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:52,065 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:52,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:52,065 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:52,065 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:52,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:52,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:52,067 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@48ec8f62[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:52,067 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:52,067 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:52,068 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:52,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:52,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:52,069 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:52,069 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:52,069 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:52,069 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,069 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:52,071 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:52,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:52,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:52,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:52,073 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:52,073 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:52,073 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:52,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:52,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:52,075 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:52,075 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:52,076 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:52,076 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:52,076 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:52,076 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:52,078 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:52,078 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:52,078 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:52,078 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:52,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:52,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:52,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:52,079 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:52,079 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:52,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:52,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:52,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,082 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:52,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,086 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:52,086 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:52,086 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:52,086 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:52,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:52,086 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:52,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:52,086 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:52,086 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:52,086 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,086 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:52,086 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:52,086 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:52,086 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:52,086 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:52,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:52,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:52,119 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:52,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922612116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,119 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:52,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922612116, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,120 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:52,120 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922612118, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,125 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:52,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 11 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922612122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,126 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:52,126 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922612123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,186 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:52,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:52,188 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:52,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:52,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:52,189 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:52,189 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:52,189 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:52,189 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:52,190 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:52,190 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:52,191 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:52,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:52,191 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:52,192 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,192 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:52,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:52,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:52,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:52,192 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:52,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:52,193 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:52,193 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:52,193 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:52,193 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:52,193 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:52,193 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:52,193 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:52,193 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:52,193 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:52,193 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:52,193 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:52,193 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,193 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:52,193 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:52,196 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:52,196 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:52,196 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:52,196 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:52,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:52,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:52,197 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a36338a[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:52,197 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:52,197 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:52,199 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:52,199 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:52,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:52,199 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,199 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:52,199 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:52,199 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:52,199 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:52,202 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:52,202 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:52,202 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,202 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:52,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:52,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:52,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:52,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:52,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:52,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:52,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:52,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:52,205 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:52,205 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:52,205 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:52,205 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:52,205 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:52,205 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:52,208 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:52,208 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:52,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:52,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:52,208 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/3499d35f07494df9a650a06206e42859 2023-04-08 02:55:52,208 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:52,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:52,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:52,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:52,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:52,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:52,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:52,208 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:52,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:52,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:52,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:52,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,210 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:52,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:52,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,213 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:52,213 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:52,213 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:52,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:52,213 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,213 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:52,213 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:52,213 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:52,213 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,214 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:52,214 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:52,214 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:52,213 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:52,214 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:52,214 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:52,214 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:52,214 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:52,214 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:52,215 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:52,220 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/3b3d85d989e64d5a991469ce5ebe9e68 2023-04-08 02:55:52,229 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=15 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/d184555db9c94742b15c4f68a6411216 2023-04-08 02:55:52,232 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/3499d35f07494df9a650a06206e42859 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/3499d35f07494df9a650a06206e42859 2023-04-08 02:55:52,236 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/3499d35f07494df9a650a06206e42859, entries=150, sequenceid=15, filesize=11.6 K 2023-04-08 02:55:52,237 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/3b3d85d989e64d5a991469ce5ebe9e68 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/3b3d85d989e64d5a991469ce5ebe9e68 2023-04-08 02:55:52,241 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/3b3d85d989e64d5a991469ce5ebe9e68, entries=150, sequenceid=15, filesize=11.6 K 2023-04-08 02:55:52,242 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/d184555db9c94742b15c4f68a6411216 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/d184555db9c94742b15c4f68a6411216 2023-04-08 02:55:52,246 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/d184555db9c94742b15c4f68a6411216, entries=150, sequenceid=15, filesize=11.6 K 2023-04-08 02:55:52,247 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=107.34 KB/109920 for 61e4c342830c067a5389bf73aa5cd767 in 456ms, sequenceid=15, compaction requested=false 2023-04-08 02:55:52,247 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:55:52,314 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:52,315 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:52,316 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:52,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:52,317 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:52,317 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:52,317 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:52,318 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:52,318 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:52,318 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:52,318 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:52,319 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,319 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:52,319 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:52,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:52,319 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,319 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:52,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:52,320 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:52,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:52,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:52,320 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:52,321 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:52,321 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:52,321 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:52,321 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:52,321 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:52,321 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:52,321 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:52,321 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:52,321 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:52,321 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-08 02:55:52,321 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:55:52,321 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:52,321 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:55:52,321 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:52,322 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:55:52,322 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:52,424 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:55:52,424 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:52,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:52,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922612431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,434 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:52,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922612431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,436 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:52,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922612432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,437 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:52,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 15 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922612432, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,437 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:52,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922612433, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:52,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922612535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,538 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:52,538 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922612535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,540 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:52,541 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922612538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,543 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:52,544 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922612538, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,545 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:52,545 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 17 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922612542, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,729 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/3e3c4c3896b34d77ad3c2f37885d406e 2023-04-08 02:55:52,738 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/cf67290693414bb68a9c3b1125452f67 2023-04-08 02:55:52,742 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:52,742 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922612740, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,744 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:52,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922612741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,745 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:52,745 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922612743, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,747 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:52,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922612745, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:52,749 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:52,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 19 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922612746, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,047 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:53,047 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922613045, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:53,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922613046, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,048 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:53,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922613047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,051 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:53,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 23 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922613048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,052 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:53,052 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 21 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922613051, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,149 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=34 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/98f8ccd9832344b9bf3188eef1098597 2023-04-08 02:55:53,153 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/3e3c4c3896b34d77ad3c2f37885d406e as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/3e3c4c3896b34d77ad3c2f37885d406e 2023-04-08 02:55:53,156 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/3e3c4c3896b34d77ad3c2f37885d406e, entries=150, sequenceid=34, filesize=11.6 K 2023-04-08 02:55:53,157 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/cf67290693414bb68a9c3b1125452f67 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/cf67290693414bb68a9c3b1125452f67 2023-04-08 02:55:53,161 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/cf67290693414bb68a9c3b1125452f67, entries=150, sequenceid=34, filesize=11.6 K 2023-04-08 02:55:53,162 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/98f8ccd9832344b9bf3188eef1098597 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/98f8ccd9832344b9bf3188eef1098597 2023-04-08 02:55:53,173 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/98f8ccd9832344b9bf3188eef1098597, entries=150, sequenceid=34, filesize=11.6 K 2023-04-08 02:55:53,174 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 61e4c342830c067a5389bf73aa5cd767 in 853ms, sequenceid=34, compaction requested=false 2023-04-08 02:55:53,174 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:55:53,174 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:53,174 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:53,174 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:53,174 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,176 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:53,176 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:53,177 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:53,178 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:53,178 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:53,179 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:53,179 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:53,180 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:53,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:53,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:53,182 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@b39e136[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:53,182 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:53,182 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:53,184 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:53,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:53,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:53,184 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:53,184 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:53,184 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:53,186 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:53,186 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:53,186 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:53,187 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,187 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:53,187 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:53,187 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:53,187 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:53,187 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:53,187 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:53,187 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:53,190 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:53,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:53,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:53,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:53,190 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:53,190 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:53,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:53,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:53,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:53,190 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:53,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:53,191 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:53,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:53,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:53,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:53,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,192 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:53,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,194 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,194 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:53,194 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:53,194 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:53,194 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:53,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:53,194 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:53,194 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:53,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:53,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:53,194 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:53,194 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:53,194 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:53,194 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,194 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:53,194 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:53,194 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:53,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:53,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:53,294 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:53,295 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:53,296 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:53,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:53,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:53,298 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:53,298 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:53,298 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:53,298 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:53,298 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:53,298 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:53,305 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,305 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:53,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:53,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:53,305 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,305 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:53,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:53,306 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:53,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:53,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:53,306 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:53,306 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:53,306 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:53,307 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:53,307 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:53,307 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:53,307 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:53,307 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:53,307 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:53,307 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:53,307 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-08 02:55:53,307 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:55:53,307 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:53,307 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:55:53,307 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:53,308 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:55:53,308 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:53,553 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:55:53,553 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:53,572 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:53,572 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 25 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922613568, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,575 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:53,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922613569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,576 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:53,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922613570, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,576 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:53,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922613571, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,577 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:53,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922613572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:53,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922613673, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:53,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922613676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,678 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:53,678 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922613676, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,680 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:53,680 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922613677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,681 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:53,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 28 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922613678, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,715 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/5e411958352b4d8b9f732523078b84ee 2023-04-08 02:55:53,724 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/29ed95cd378d40ffb17a4dd671e1a5bb 2023-04-08 02:55:53,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:53,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922613879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,881 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:53,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922613879, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,882 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:53,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922613880, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,883 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:53,883 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922613881, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:53,885 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:53,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 30 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922613882, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,133 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=47 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/eb7ca69ac73a4d219ac10845578bf0e3 2023-04-08 02:55:54,138 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/5e411958352b4d8b9f732523078b84ee as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/5e411958352b4d8b9f732523078b84ee 2023-04-08 02:55:54,145 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/5e411958352b4d8b9f732523078b84ee, entries=150, sequenceid=47, filesize=11.6 K 2023-04-08 02:55:54,145 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/29ed95cd378d40ffb17a4dd671e1a5bb as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/29ed95cd378d40ffb17a4dd671e1a5bb 2023-04-08 02:55:54,149 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/29ed95cd378d40ffb17a4dd671e1a5bb, entries=150, sequenceid=47, filesize=11.6 K 2023-04-08 02:55:54,149 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/eb7ca69ac73a4d219ac10845578bf0e3 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/eb7ca69ac73a4d219ac10845578bf0e3 2023-04-08 02:55:54,153 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/eb7ca69ac73a4d219ac10845578bf0e3, entries=150, sequenceid=47, filesize=11.6 K 2023-04-08 02:55:54,153 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 61e4c342830c067a5389bf73aa5cd767 in 846ms, sequenceid=47, compaction requested=true 2023-04-08 02:55:54,153 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:55:54,154 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:54,154 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:54,154 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:54,154 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,154 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:54,154 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:54,155 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,155 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:54,156 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,156 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:54,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:54,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:54,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26a2d8c5[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:54,157 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:54,157 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,159 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,159 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:54,159 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:54,159 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:54,159 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,159 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:54,160 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:54,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:54,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:54,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:54,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:54,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:54,163 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:54,163 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:54,163 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:54,163 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:54,163 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:54,163 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:54,167 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,168 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:54,168 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:54,168 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:54,168 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,168 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,168 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:54,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,168 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:54,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:54,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:54,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:54,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:54,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:54,173 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:54,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,175 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,175 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:54,175 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:54,175 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:54,175 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:54,176 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:54,175 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:54,176 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:54,176 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:54,175 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:54,176 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,176 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,176 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,176 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:54,176 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:54,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:54,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:54,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:55:54,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680922553569, max timestamp=1680922554188Num uniques 859; 2023-04-08 02:55:54,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-08 02:55:54,190 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680922553569, max timestamp=1680922554188Num uniques 859; 2023-04-08 02:55:54,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-08 02:55:54,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680922553569, max timestamp=1680922554188Num uniques 859; 2023-04-08 02:55:54,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-08 02:55:54,191 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=55.91 KB heapSize=154.41 KB 2023-04-08 02:55:54,192 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:55:54,192 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:54,192 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:55:54,192 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:54,192 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:55:54,192 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:54,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680922554211, max timestamp=1680922554223Num uniques 859; 2023-04-08 02:55:54,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922554217, max timestamp=1680922554223Num uniques 859; 2023-04-08 02:55:54,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:55:54,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-08 02:55:54,225 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922554217, max timestamp=1680922554223Num uniques 859; 2023-04-08 02:55:54,226 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:55:54,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:54,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922614254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:54,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922614254, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,262 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:54,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922614257, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:54,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922614262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,269 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:54,269 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922614262, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,276 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:54,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:54,280 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:54,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:54,281 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:54,281 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:54,281 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:54,281 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:54,281 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:54,282 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,282 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:54,283 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,283 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:54,283 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:54,283 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:54,284 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,284 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,284 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:54,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:54,285 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:54,285 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:54,285 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:54,285 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:54,285 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:54,285 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:54,285 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:54,285 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:54,285 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:54,285 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:54,285 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:54,286 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:54,286 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:54,286 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:54,286 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,286 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:54,286 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:54,287 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,288 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:54,288 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,289 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,289 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:54,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:54,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:54,290 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a19c4ec[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:54,290 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:54,290 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,292 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,292 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,293 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,293 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:54,293 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:54,293 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:54,293 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:54,294 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,294 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:54,294 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:54,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:54,294 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:54,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:54,296 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:54,297 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:54,299 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:54,299 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:54,299 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:54,299 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:54,299 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:54,304 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,304 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:54,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,304 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:54,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:54,305 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:54,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,305 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:54,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,307 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:54,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,310 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,310 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:54,310 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:54,310 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:54,310 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,310 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:54,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:54,310 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,310 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,310 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:54,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:54,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:54,310 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:54,310 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:54,310 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:54,310 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:54,310 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:54,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:54,369 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:54,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922614363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:54,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922614363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,370 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:54,370 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922614363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,378 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:54,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 41 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922614370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,378 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:54,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922614370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,410 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:54,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:54,413 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:54,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:54,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:54,422 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:54,422 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:54,422 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:54,422 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:54,423 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,423 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:54,424 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,424 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:54,424 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:54,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:54,424 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,424 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:54,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:54,425 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:54,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:54,425 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:54,426 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:54,426 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:54,426 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:54,426 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:54,426 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:54,428 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:54,428 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:54,428 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:54,428 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:54,428 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:54,428 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:54,428 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:54,428 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:54,429 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,429 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:54,429 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:54,430 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,430 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:54,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:54,431 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,431 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:54,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:54,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:54,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:54,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:54,434 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b81c46[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:54,434 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:54,434 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,436 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,436 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:54,436 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:54,436 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:54,436 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,436 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:54,443 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:54,443 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:54,443 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:54,444 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:54,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:54,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:54,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:54,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:54,446 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:54,446 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:54,446 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:54,446 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:54,447 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:54,450 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:54,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:54,450 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,451 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:54,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:54,451 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:54,451 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,453 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,455 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:54,455 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:54,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:54,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:54,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:54,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:54,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:54,456 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,456 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:54,456 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:54,456 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,456 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,456 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,456 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,456 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:54,456 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:54,457 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:54,458 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:54,458 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:54,558 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:54,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:54,561 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:54,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:54,562 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:54,562 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:54,562 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:54,562 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:54,562 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:54,563 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,563 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:54,565 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:54,565 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,565 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:54,565 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,565 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:54,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:54,566 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:54,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:54,566 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:54,566 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:54,567 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:54,567 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:54,567 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:54,567 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:54,567 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:54,567 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:54,567 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:54,567 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:54,567 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:54,567 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:54,567 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:54,567 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:54,567 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,567 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:54,567 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:54,569 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,569 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:54,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:54,570 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,570 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:54,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:54,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:54,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:54,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:54,571 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@454c6d16[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:54,571 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:54,572 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,573 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,573 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:54,573 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:54,573 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:54,574 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,574 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:54,575 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,575 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:54,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:54,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:54,575 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:54,575 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:54,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:54,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:54,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:54,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:54,578 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:54,578 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:54,578 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:54,578 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:54,578 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:54,578 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:54,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:54,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922614572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:54,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922614572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:54,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922614573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,581 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,581 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,581 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:54,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:54,581 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:54,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:54,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:54,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:54,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,583 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:54,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,584 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:54,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:54,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 43 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922614580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,585 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:54,585 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922614580, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,588 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,588 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:54,588 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:54,588 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:54,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:54,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:54,588 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:54,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:54,588 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:54,588 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:54,588 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,588 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,588 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,588 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,588 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:54,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:54,588 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:54,589 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:54,603 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/6a42ee5ab47e4a91bf59c36e3cd55d85 2023-04-08 02:55:54,631 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/ac4a54b690fb40f48070ba4f01a3d689 2023-04-08 02:55:54,664 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=70 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/28054d378aff4665832b7c46bfe3ffe1 2023-04-08 02:55:54,669 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/6a42ee5ab47e4a91bf59c36e3cd55d85 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/6a42ee5ab47e4a91bf59c36e3cd55d85 2023-04-08 02:55:54,683 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/6a42ee5ab47e4a91bf59c36e3cd55d85, entries=250, sequenceid=70, filesize=16.2 K 2023-04-08 02:55:54,684 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/ac4a54b690fb40f48070ba4f01a3d689 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/ac4a54b690fb40f48070ba4f01a3d689 2023-04-08 02:55:54,687 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/ac4a54b690fb40f48070ba4f01a3d689, entries=150, sequenceid=70, filesize=11.6 K 2023-04-08 02:55:54,688 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/28054d378aff4665832b7c46bfe3ffe1 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/28054d378aff4665832b7c46bfe3ffe1 2023-04-08 02:55:54,689 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:54,689 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:54,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:54,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:54,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:54,693 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:54,693 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:54,693 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:54,693 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:54,693 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,693 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:54,694 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,694 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:54,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:54,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:54,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:54,695 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:54,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:54,695 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:54,695 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,695 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:54,696 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:54,696 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:54,696 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:54,696 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:54,696 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:54,701 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/28054d378aff4665832b7c46bfe3ffe1, entries=150, sequenceid=70, filesize=11.6 K 2023-04-08 02:55:54,701 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:54,702 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:54,702 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~121.27 KB/124176, currentSize=138.65 KB/141980 for 61e4c342830c067a5389bf73aa5cd767 in 511ms, sequenceid=70, compaction requested=true 2023-04-08 02:55:54,702 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:54,702 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:54,702 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:55:54,702 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=138.65 KB heapSize=395.14 KB 2023-04-08 02:55:54,702 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:55:54,702 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-08 02:55:54,702 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:54,702 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:55:54,703 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:54,703 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:54,703 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 52224 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:55:54,703 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/A is initiating minor compaction (all files) 2023-04-08 02:55:54,704 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/A in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:54,704 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/3499d35f07494df9a650a06206e42859, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/3e3c4c3896b34d77ad3c2f37885d406e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/5e411958352b4d8b9f732523078b84ee, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/6a42ee5ab47e4a91bf59c36e3cd55d85] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=51 K 2023-04-08 02:55:54,704 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:55:54,704 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:54,704 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 3499d35f07494df9a650a06206e42859, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680922551785 2023-04-08 02:55:54,704 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:55:54,704 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:55:54,704 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/B is initiating minor compaction (all files) 2023-04-08 02:55:54,704 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:54,704 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/B in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:54,704 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 3e3c4c3896b34d77ad3c2f37885d406e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1680922551809 2023-04-08 02:55:54,705 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/3b3d85d989e64d5a991469ce5ebe9e68, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/cf67290693414bb68a9c3b1125452f67, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/29ed95cd378d40ffb17a4dd671e1a5bb, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/ac4a54b690fb40f48070ba4f01a3d689] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=46.4 K 2023-04-08 02:55:54,705 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:55:54,705 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:54,705 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 5e411958352b4d8b9f732523078b84ee, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680922552430 2023-04-08 02:55:54,705 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 3b3d85d989e64d5a991469ce5ebe9e68, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680922551785 2023-04-08 02:55:54,705 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 6a42ee5ab47e4a91bf59c36e3cd55d85, keycount=250, bloomtype=ROW, size=16.2 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1680922553569 2023-04-08 02:55:54,705 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting cf67290693414bb68a9c3b1125452f67, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1680922551809 2023-04-08 02:55:54,706 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 29ed95cd378d40ffb17a4dd671e1a5bb, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680922552430 2023-04-08 02:55:54,706 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting ac4a54b690fb40f48070ba4f01a3d689, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1680922553571 2023-04-08 02:55:54,744 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#B#compaction#465 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:54,752 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#A#compaction#466 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:54,809 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=49.20 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/93a402755947436b95f5f9760bc351a8 2023-04-08 02:55:54,813 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/241f943f8417466eb85ab943733ab161 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/241f943f8417466eb85ab943733ab161 2023-04-08 02:55:54,815 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/f914c13f67f14beabbb64f26638a722e as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/f914c13f67f14beabbb64f26638a722e 2023-04-08 02:55:54,819 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/B of 61e4c342830c067a5389bf73aa5cd767 into f914c13f67f14beabbb64f26638a722e(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:54,819 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:55:54,819 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/B, priority=12, startTime=1680922554702; duration=0sec 2023-04-08 02:55:54,819 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:54,819 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:55:54,820 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/A of 61e4c342830c067a5389bf73aa5cd767 into 241f943f8417466eb85ab943733ab161(size=11.7 K), total size for store is 11.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:54,820 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:55:54,820 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/A, priority=12, startTime=1680922554702; duration=0sec 2023-04-08 02:55:54,820 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:54,821 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47544 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:55:54,821 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/C is initiating minor compaction (all files) 2023-04-08 02:55:54,821 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/C in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:54,821 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/d184555db9c94742b15c4f68a6411216, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/98f8ccd9832344b9bf3188eef1098597, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/eb7ca69ac73a4d219ac10845578bf0e3, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/28054d378aff4665832b7c46bfe3ffe1] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=46.4 K 2023-04-08 02:55:54,821 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting d184555db9c94742b15c4f68a6411216, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=15, earliestPutTs=1680922551785 2023-04-08 02:55:54,822 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 98f8ccd9832344b9bf3188eef1098597, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=34, earliestPutTs=1680922551809 2023-04-08 02:55:54,822 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting eb7ca69ac73a4d219ac10845578bf0e3, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=47, earliestPutTs=1680922552430 2023-04-08 02:55:54,822 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 28054d378aff4665832b7c46bfe3ffe1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1680922553571 2023-04-08 02:55:54,824 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/ab4659ccb1a14712af3748beb99cae4a 2023-04-08 02:55:54,829 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#C#compaction#468 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:54,837 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=107 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/87c0cc3d7db743d8835402c8a5edefe0 2023-04-08 02:55:54,842 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/93a402755947436b95f5f9760bc351a8 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/93a402755947436b95f5f9760bc351a8 2023-04-08 02:55:54,846 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/93a402755947436b95f5f9760bc351a8, entries=150, sequenceid=107, filesize=11.6 K 2023-04-08 02:55:54,846 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/ab4659ccb1a14712af3748beb99cae4a as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/ab4659ccb1a14712af3748beb99cae4a 2023-04-08 02:55:54,851 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/ab4659ccb1a14712af3748beb99cae4a, entries=150, sequenceid=107, filesize=11.6 K 2023-04-08 02:55:54,857 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/87c0cc3d7db743d8835402c8a5edefe0 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/87c0cc3d7db743d8835402c8a5edefe0 2023-04-08 02:55:54,861 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/87c0cc3d7db743d8835402c8a5edefe0, entries=150, sequenceid=107, filesize=11.6 K 2023-04-08 02:55:54,861 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~138.65 KB/141980, heapSize ~395.09 KB/404576, currentSize=0 B/0 for 61e4c342830c067a5389bf73aa5cd767 in 159ms, sequenceid=107, compaction requested=false 2023-04-08 02:55:54,861 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:55:54,861 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:54,861 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:54,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:54,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:54,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:54,865 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:54,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:54,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,866 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:54,866 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,866 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:54,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7598f18a[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:54,867 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:54,867 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,867 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,867 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:54,870 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,870 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,870 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:54,871 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:54,871 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:54,871 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,871 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:54,872 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:54,872 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:54,872 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:54,873 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:54,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:54,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:54,877 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:54,877 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:54,877 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:54,877 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:54,877 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:54,877 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:54,879 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,879 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,879 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:54,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,879 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:54,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:54,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:54,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,882 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:54,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:54,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,888 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,888 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:54,888 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:54,888 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:54,888 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:54,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:54,888 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:54,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:54,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:54,888 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:54,888 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,888 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,888 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:54,888 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,888 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:54,888 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:54,888 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:54,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:54,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:54,893 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:55:54,893 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-08 02:55:54,893 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:55:54,893 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:54,893 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:55:54,893 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:54,893 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:55:54,893 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:54,920 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:54,920 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922614916, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680922554907, max timestamp=1680922554919Num uniques 859; 2023-04-08 02:55:54,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-08 02:55:54,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680922554907, max timestamp=1680922554919Num uniques 859; 2023-04-08 02:55:54,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-08 02:55:54,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=309, cellSize=14145, totalHeapSize=32376, min timestamp=1680922554907, max timestamp=1680922554919Num uniques 859; 2023-04-08 02:55:54,921 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=14145, suffix heap size=117064,new segment heap size=32376 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=309 2023-04-08 02:55:54,954 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:54,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922614948, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,956 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:54,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922614952, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,961 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:54,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922614953, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,962 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:54,962 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 54 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922614954, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,988 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:54,989 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:54,991 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:54,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:54,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:54,992 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:54,992 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:54,992 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:54,992 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:54,993 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,993 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:54,994 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,994 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:54,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:54,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:54,995 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,995 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:54,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:54,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:54,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:54,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:54,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:54,996 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:54,996 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:54,996 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:54,996 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:54,996 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:55,000 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:55,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:55,000 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:55,000 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,000 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:55,000 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:55,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:55,001 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@31e27571[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:55,001 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:55,002 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,004 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,004 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,004 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,004 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:55,004 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:55,004 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:55,004 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,004 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:55,006 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,006 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:55,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:55,007 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:55,007 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:55,007 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:55,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:55,007 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:55,008 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:55,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:55,010 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:55,010 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:55,010 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:55,010 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:55,010 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:55,012 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,012 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:55,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:55,012 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:55,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,012 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:55,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:55,012 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:55,013 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:55,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,014 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,016 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:55,016 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:55,016 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:55,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:55,016 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,016 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:55,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:55,016 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:55,016 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:55,016 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:55,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:55,017 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:55,017 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,017 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,017 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,017 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,017 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,017 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:55,032 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:55,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922615029, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:55,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922615055, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,063 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:55,063 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922615057, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:55,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922615062, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:55,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 56 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922615063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,117 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:55,118 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:55,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:55,123 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:55,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:55,124 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:55,124 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:55,124 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:55,124 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:55,124 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,124 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,126 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:55,126 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,126 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:55,127 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,127 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,127 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:55,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,127 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:55,128 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:55,128 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:55,128 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:55,128 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:55,128 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:55,132 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:55,132 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:55,132 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:55,132 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:55,132 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:55,132 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:55,132 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:55,132 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:55,132 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,132 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:55,132 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:55,134 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,134 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:55,134 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,134 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:55,134 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,135 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:55,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:55,135 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70f91527[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:55,135 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:55,135 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,136 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,136 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,136 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:55,136 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,136 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:55,136 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:55,137 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:55,138 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,138 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:55,138 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:55,138 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:55,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:55,140 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:55,144 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:55,145 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:55,147 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:55,147 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:55,147 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:55,147 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:55,147 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:55,147 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:55,150 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,150 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,150 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:55,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:55,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:55,156 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:55,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:55,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:55,156 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,157 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,159 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,159 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,159 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,159 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,159 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,159 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,159 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,159 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:55,159 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:55,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:55,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:55,159 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:55,159 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:55,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:55,160 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:55,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:55,160 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:55,160 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:55,160 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:55,237 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/4fbb3e6acd104490ab3fa3d4c265cd25 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/4fbb3e6acd104490ab3fa3d4c265cd25 2023-04-08 02:55:55,241 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/C of 61e4c342830c067a5389bf73aa5cd767 into 4fbb3e6acd104490ab3fa3d4c265cd25(size=11.7 K), total size for store is 23.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:55,241 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:55:55,242 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/C, priority=12, startTime=1680922554703; duration=0sec 2023-04-08 02:55:55,242 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:55,242 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:55,242 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 55 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922615234, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,260 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:55,261 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:55,262 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:55,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:55,263 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:55,263 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:55,263 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:55,263 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:55,263 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:55,264 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,264 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,266 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,266 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:55,266 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:55,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:55,266 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,266 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:55,266 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,267 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:55,267 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,267 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:55,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:55,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922615265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:55,274 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 59 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922615265, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,274 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:55,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 58 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922615267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,275 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:55,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922615267, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,280 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:55,280 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:55,280 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:55,280 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:55,280 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:55,288 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:55,288 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:55,288 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:55,288 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:55,288 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:55,288 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:55,288 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:55,288 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:55,288 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,288 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:55,288 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:55,290 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,290 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,290 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:55,291 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:55,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:55,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:55,293 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56fbf2f3[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:55,293 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:55,293 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,295 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,295 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:55,295 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:55,295 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:55,296 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,296 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:55,297 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:55,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:55,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:55,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:55,298 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:55,298 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:55,298 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:55,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:55,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,300 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:55,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:55,300 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:55,300 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:55,300 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:55,300 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:55,300 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:55,302 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,302 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,302 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:55,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:55,303 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:55,303 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/03f06a5c2a4949f6a33fbfff6ca77d8e 2023-04-08 02:55:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:55,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,304 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:55,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,307 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:55,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:55,307 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:55,307 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:55,307 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:55,307 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:55,307 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:55,307 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,308 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,308 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,308 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,308 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,308 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,308 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,307 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:55,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:55,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:55,308 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:55,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:55,408 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:55,408 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:55,410 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:55,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:55,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:55,411 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:55,411 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:55,411 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:55,411 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:55,412 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,412 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,419 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,419 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:55,419 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:55,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:55,420 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,420 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:55,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,420 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:55,420 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,420 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:55,421 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:55,421 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:55,421 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:55,421 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:55,421 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:55,421 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:55,422 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:55,422 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:55,422 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:55,422 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:55,422 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:55,422 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:55,422 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:55,422 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,422 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:55,422 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:55,423 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,423 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:55,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:55,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:55,424 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,424 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:55,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:55,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:55,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:55,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@64dd5d07[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:55,425 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:55,425 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,426 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,426 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,426 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:55,427 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:55,427 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:55,427 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,427 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:55,430 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:55,430 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:55,430 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,430 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:55,430 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:55,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:55,433 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:55,433 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:55,433 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:55,433 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:55,433 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:55,435 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,435 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,435 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:55,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:55,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:55,436 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,437 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,439 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,439 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:55,439 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:55,439 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,439 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:55,439 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:55,439 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:55,439 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,440 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,440 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,440 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:55,439 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:55,440 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:55,440 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:55,440 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,440 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:55,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:55,540 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:55,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:55,555 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:55,557 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:55,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 57 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922615552, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:55,557 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:55,558 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:55,558 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:55,558 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:55,558 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:55,558 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,558 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,560 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,560 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:55,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:55,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:55,561 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,561 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:55,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,562 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:55,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:55,563 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:55,563 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:55,563 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:55,563 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:55,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:55,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:55,564 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:55,564 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:55,564 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:55,564 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:55,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:55,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:55,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:55,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:55,567 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,567 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,567 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:55,568 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:55,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:55,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:55,569 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4636f510[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:55,569 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:55,570 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,571 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,571 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,571 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:55,571 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:55,571 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:55,571 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,571 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:55,572 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:55,572 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:55,572 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,572 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:55,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:55,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:55,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:55,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:55,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:55,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,575 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:55,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922615575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:55,577 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:55,577 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:55,577 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:55,577 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:55,577 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:55,577 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:55,579 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,579 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:55,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:55,579 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,579 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:55,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:55,580 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:55,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:55,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,582 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:55,582 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922615576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,582 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:55,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:55,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922615577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,584 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:55,584 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922615578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,588 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:55,588 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:55,588 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,589 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:55,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:55,589 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:55,589 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:55,589 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,589 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,589 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:55,589 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:55,589 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:55,589 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,589 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:55,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:55,590 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:55,689 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:55,690 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:55,691 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:55,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:55,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:55,693 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:55,693 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:55,693 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:55,693 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:55,694 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,694 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,696 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:55,696 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:55,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:55,696 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,696 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,696 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:55,697 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:55,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:55,697 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:55,697 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:55,697 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:55,697 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:55,697 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:55,700 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:55,700 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:55,700 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:55,700 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:55,700 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:55,700 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:55,700 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:55,700 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:55,700 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,700 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:55,700 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:55,704 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,704 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:55,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:55,705 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,705 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:55,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:55,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:55,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7e5e80dc[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:55,706 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:55,706 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,709 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,709 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:55,709 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:55,709 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:55,709 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,710 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:55,712 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:55,712 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:55,712 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:55,712 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:55,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:55,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:55,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:55,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,718 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/2fbb7de3332e4dd293ec407b243380ce 2023-04-08 02:55:55,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:55,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:55,721 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:55,721 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:55,721 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:55,721 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:55,721 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:55,721 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:55,725 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:55,725 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,725 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:55,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,726 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:55,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:55,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,728 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:55,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,729 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:55,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:55,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,735 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,735 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,735 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,735 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,735 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,735 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:55,735 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,735 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:55,736 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:55,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:55,736 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:55,736 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:55,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:55,736 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:55,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:55,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:55,736 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:55,736 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:55,754 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=120 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/4cb12aec980d4d05bc22657e5f62dda1 2023-04-08 02:55:55,763 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/03f06a5c2a4949f6a33fbfff6ca77d8e as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/03f06a5c2a4949f6a33fbfff6ca77d8e 2023-04-08 02:55:55,767 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/03f06a5c2a4949f6a33fbfff6ca77d8e, entries=200, sequenceid=120, filesize=13.9 K 2023-04-08 02:55:55,768 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/2fbb7de3332e4dd293ec407b243380ce as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/2fbb7de3332e4dd293ec407b243380ce 2023-04-08 02:55:55,772 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/2fbb7de3332e4dd293ec407b243380ce, entries=150, sequenceid=120, filesize=11.6 K 2023-04-08 02:55:55,772 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/4cb12aec980d4d05bc22657e5f62dda1 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/4cb12aec980d4d05bc22657e5f62dda1 2023-04-08 02:55:55,783 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/4cb12aec980d4d05bc22657e5f62dda1, entries=150, sequenceid=120, filesize=11.6 K 2023-04-08 02:55:55,783 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=134.18 KB/137400 for 61e4c342830c067a5389bf73aa5cd767 in 890ms, sequenceid=120, compaction requested=true 2023-04-08 02:55:55,783 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:55:55,784 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:55,784 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:55,784 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:55:55,784 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:55:55,784 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:55,784 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38136 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:55:55,785 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/A is initiating minor compaction (all files) 2023-04-08 02:55:55,785 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/A in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:55,785 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/241f943f8417466eb85ab943733ab161, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/93a402755947436b95f5f9760bc351a8, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/03f06a5c2a4949f6a33fbfff6ca77d8e] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=37.2 K 2023-04-08 02:55:55,785 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 241f943f8417466eb85ab943733ab161, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1680922553571 2023-04-08 02:55:55,785 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:55:55,785 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/B is initiating minor compaction (all files) 2023-04-08 02:55:55,785 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 93a402755947436b95f5f9760bc351a8, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680922554253 2023-04-08 02:55:55,786 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/B in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:55,786 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/f914c13f67f14beabbb64f26638a722e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/ab4659ccb1a14712af3748beb99cae4a, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/2fbb7de3332e4dd293ec407b243380ce] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=35.0 K 2023-04-08 02:55:55,786 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 03f06a5c2a4949f6a33fbfff6ca77d8e, keycount=200, bloomtype=ROW, size=13.9 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680922554890 2023-04-08 02:55:55,786 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting f914c13f67f14beabbb64f26638a722e, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1680922553571 2023-04-08 02:55:55,787 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting ab4659ccb1a14712af3748beb99cae4a, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680922554253 2023-04-08 02:55:55,787 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 2fbb7de3332e4dd293ec407b243380ce, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680922554890 2023-04-08 02:55:55,793 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#A#compaction#473 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:55,795 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#B#compaction#474 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:55,836 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:55,837 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:55,839 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:55,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:55,840 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:55,840 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:55,840 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:55,840 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:55,840 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:55,841 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,841 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,842 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:55,842 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:55,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:55,843 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,843 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:55,845 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:55,845 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:55,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:55,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:55,845 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:55,856 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:55,856 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:55,860 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:55,860 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:55,860 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:55,868 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:55,868 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:55,868 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:55,868 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:55,868 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=134.18 KB heapSize=370.92 KB 2023-04-08 02:55:55,868 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:55:55,868 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:55,869 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:55:55,869 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:55,869 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:55:55,869 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:55:56,069 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:56,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:55:56,097 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:56,097 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922616088, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,100 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:56,100 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922616089, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,105 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:56,105 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922616096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,106 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:56,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922616096, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,106 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:56,106 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922616097, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:56,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922616203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,213 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/b0ec3fd0501d49a2bc3bda12e58f0c46 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/b0ec3fd0501d49a2bc3bda12e58f0c46 2023-04-08 02:55:56,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:56,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922616206, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:56,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922616207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:56,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922616207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,219 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/A of 61e4c342830c067a5389bf73aa5cd767 into b0ec3fd0501d49a2bc3bda12e58f0c46(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:56,219 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:55:56,219 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/A, priority=13, startTime=1680922555784; duration=0sec 2023-04-08 02:55:56,219 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:56,219 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:55:56,220 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35796 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:55:56,220 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/C is initiating minor compaction (all files) 2023-04-08 02:55:56,220 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/C in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:56,220 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/4fbb3e6acd104490ab3fa3d4c265cd25, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/87c0cc3d7db743d8835402c8a5edefe0, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/4cb12aec980d4d05bc22657e5f62dda1] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=35.0 K 2023-04-08 02:55:56,221 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 4fbb3e6acd104490ab3fa3d4c265cd25, keycount=150, bloomtype=ROW, size=11.7 K, encoding=NONE, compression=NONE, seqNum=70, earliestPutTs=1680922553571 2023-04-08 02:55:56,221 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 87c0cc3d7db743d8835402c8a5edefe0, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=107, earliestPutTs=1680922554253 2023-04-08 02:55:56,221 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 4cb12aec980d4d05bc22657e5f62dda1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680922554890 2023-04-08 02:55:56,227 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#C#compaction#476 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:56,230 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/44f1bb9bc5d749a5bbbe625f6b0d5ae9 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/44f1bb9bc5d749a5bbbe625f6b0d5ae9 2023-04-08 02:55:56,234 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/B of 61e4c342830c067a5389bf73aa5cd767 into 44f1bb9bc5d749a5bbbe625f6b0d5ae9(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:56,234 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:55:56,234 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/B, priority=13, startTime=1680922555784; duration=0sec 2023-04-08 02:55:56,234 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:56,328 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/dd41141929924724aede4688c6e45ee4 2023-04-08 02:55:56,340 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/83c84fa075f943f3b7e6e6d9707b079d 2023-04-08 02:55:56,352 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=155 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/beae83dc729e4325bd17f230aa6dea58 2023-04-08 02:55:56,361 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/dd41141929924724aede4688c6e45ee4 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/dd41141929924724aede4688c6e45ee4 2023-04-08 02:55:56,366 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/dd41141929924724aede4688c6e45ee4, entries=150, sequenceid=155, filesize=11.8 K 2023-04-08 02:55:56,367 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/83c84fa075f943f3b7e6e6d9707b079d as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/83c84fa075f943f3b7e6e6d9707b079d 2023-04-08 02:55:56,374 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/83c84fa075f943f3b7e6e6d9707b079d, entries=150, sequenceid=155, filesize=11.8 K 2023-04-08 02:55:56,375 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/beae83dc729e4325bd17f230aa6dea58 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/beae83dc729e4325bd17f230aa6dea58 2023-04-08 02:55:56,378 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/beae83dc729e4325bd17f230aa6dea58, entries=150, sequenceid=155, filesize=11.8 K 2023-04-08 02:55:56,379 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~370.88 KB/379776, currentSize=53.67 KB/54960 for 61e4c342830c067a5389bf73aa5cd767 in 511ms, sequenceid=155, compaction requested=false 2023-04-08 02:55:56,379 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:55:56,379 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:56,379 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:56,379 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:56,379 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,379 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:56,379 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:56,384 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,384 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:56,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:56,385 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,385 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:56,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:56,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:56,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:56,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:56,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:56,389 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@773b898e[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:56,389 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:56,389 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,392 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,392 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,392 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:56,392 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,392 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:56,392 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:56,392 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:56,394 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:56,394 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:56,394 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,394 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:56,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:56,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:56,397 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:56,397 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:56,397 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:56,397 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:56,397 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:56,397 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:56,398 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,398 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:56,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,398 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,398 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:56,398 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:56,399 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:56,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:56,400 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:56,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:56,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,402 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,402 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:56,402 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:56,402 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,402 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:56,402 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:56,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:56,402 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:56,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:56,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:56,402 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:56,402 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,402 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:56,402 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,402 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:56,402 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,402 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:56,402 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:56,402 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:56,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:55:56,411 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-08 02:55:56,411 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:55:56,412 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:56,412 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:55:56,412 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:56,412 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:55:56,412 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:56,432 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/c49865cc4fa74b179b236adbc2e9eb02 2023-04-08 02:55:56,442 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/80d87c3dfd4c4b2ead0d5f078da49b8b 2023-04-08 02:55:56,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:56,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922616446, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,452 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:56,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:56,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922616447, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922616450, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,452 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:56,453 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922616451, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,472 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=169 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/ef755e07c51b437a9ae99813825a8ca8 2023-04-08 02:55:56,476 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/c49865cc4fa74b179b236adbc2e9eb02 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/c49865cc4fa74b179b236adbc2e9eb02 2023-04-08 02:55:56,487 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/c49865cc4fa74b179b236adbc2e9eb02, entries=200, sequenceid=169, filesize=14.1 K 2023-04-08 02:55:56,488 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/80d87c3dfd4c4b2ead0d5f078da49b8b as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/80d87c3dfd4c4b2ead0d5f078da49b8b 2023-04-08 02:55:56,492 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/80d87c3dfd4c4b2ead0d5f078da49b8b, entries=150, sequenceid=169, filesize=11.8 K 2023-04-08 02:55:56,493 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/ef755e07c51b437a9ae99813825a8ca8 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/ef755e07c51b437a9ae99813825a8ca8 2023-04-08 02:55:56,496 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/ef755e07c51b437a9ae99813825a8ca8, entries=150, sequenceid=169, filesize=11.8 K 2023-04-08 02:55:56,497 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 61e4c342830c067a5389bf73aa5cd767 in 86ms, sequenceid=169, compaction requested=true 2023-04-08 02:55:56,497 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:55:56,498 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:55:56,498 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:56,498 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:56,498 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-08 02:55:56,500 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38588 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:55:56,500 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/A is initiating minor compaction (all files) 2023-04-08 02:55:56,500 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/A in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:56,500 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/b0ec3fd0501d49a2bc3bda12e58f0c46, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/dd41141929924724aede4688c6e45ee4, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/c49865cc4fa74b179b236adbc2e9eb02] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=37.7 K 2023-04-08 02:55:56,501 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting b0ec3fd0501d49a2bc3bda12e58f0c46, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680922554890 2023-04-08 02:55:56,501 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting dd41141929924724aede4688c6e45ee4, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680922554947 2023-04-08 02:55:56,501 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting c49865cc4fa74b179b236adbc2e9eb02, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680922556093 2023-04-08 02:55:56,502 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:56,509 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#A#compaction#482 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:56,510 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:56,514 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:56,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:56,518 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:56,518 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:56,518 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:56,518 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:56,519 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:56,519 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,519 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:56,520 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,521 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,521 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:56,522 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:56,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:56,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:56,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:56,528 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:56,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:56,528 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:56,535 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:56,535 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:56,535 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:56,535 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:56,535 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:56,536 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:56,536 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:56,536 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:56,536 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:56,536 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-08 02:55:56,536 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:55:56,537 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:56,537 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:55:56,537 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:56,537 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:55:56,537 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:56,558 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:55:56,558 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:56,566 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/4756e463510f42b0ab4c20bccb76d650 2023-04-08 02:55:56,574 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:56,574 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922616569, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:56,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922616572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:56,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922616572, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,583 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:56,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922616573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,584 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/eb4cbe428e0b4dd382eefc013ee7794e 2023-04-08 02:55:56,604 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=189 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/4369ebd17a6745c7963808cf038f470f 2023-04-08 02:55:56,609 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/4756e463510f42b0ab4c20bccb76d650 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/4756e463510f42b0ab4c20bccb76d650 2023-04-08 02:55:56,613 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/4756e463510f42b0ab4c20bccb76d650, entries=150, sequenceid=189, filesize=11.8 K 2023-04-08 02:55:56,614 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/eb4cbe428e0b4dd382eefc013ee7794e as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/eb4cbe428e0b4dd382eefc013ee7794e 2023-04-08 02:55:56,617 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/eb4cbe428e0b4dd382eefc013ee7794e, entries=150, sequenceid=189, filesize=11.8 K 2023-04-08 02:55:56,618 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/4369ebd17a6745c7963808cf038f470f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/4369ebd17a6745c7963808cf038f470f 2023-04-08 02:55:56,621 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/4369ebd17a6745c7963808cf038f470f, entries=150, sequenceid=189, filesize=11.8 K 2023-04-08 02:55:56,622 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 61e4c342830c067a5389bf73aa5cd767 in 86ms, sequenceid=189, compaction requested=true 2023-04-08 02:55:56,622 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:55:56,622 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:56,622 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:56,622 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:56,622 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,624 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:56,624 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:56,625 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,625 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:56,626 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,626 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:56,626 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:56,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:56,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:56,627 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7900d70b[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:56,627 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:56,627 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,631 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,631 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:56,632 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:56,632 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:56,634 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,634 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:56,634 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:56,634 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:56,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:56,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:56,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:56,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:56,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:56,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:56,636 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,636 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:56,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:56,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:56,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:56,637 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:56,637 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:56,637 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:56,637 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:56,637 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:56,641 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/31bad16a37854911909214542dc51809 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/31bad16a37854911909214542dc51809 2023-04-08 02:55:56,646 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,646 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,646 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:56,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:56,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:56,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:56,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,647 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:56,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:56,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:56,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,648 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:56,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:56,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,652 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/C of 61e4c342830c067a5389bf73aa5cd767 into 31bad16a37854911909214542dc51809(size=11.8 K), total size for store is 47.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:56,652 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:55:56,652 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/C, priority=13, startTime=1680922555784; duration=0sec 2023-04-08 02:55:56,652 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-08 02:55:56,652 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:55:56,652 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:56,652 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:56,652 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:56,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:56,652 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,652 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:56,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:56,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:56,653 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:56,652 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:56,653 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:56,653 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:56,653 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:56,653 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,653 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,653 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,653 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,653 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:56,653 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:56,654 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48234 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:55:56,654 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/C is initiating minor compaction (all files) 2023-04-08 02:55:56,654 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/C in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:56,654 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/31bad16a37854911909214542dc51809, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/beae83dc729e4325bd17f230aa6dea58, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/ef755e07c51b437a9ae99813825a8ca8, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/4369ebd17a6745c7963808cf038f470f] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=47.1 K 2023-04-08 02:55:56,654 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 31bad16a37854911909214542dc51809, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680922554890 2023-04-08 02:55:56,655 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting beae83dc729e4325bd17f230aa6dea58, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680922554947 2023-04-08 02:55:56,655 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting ef755e07c51b437a9ae99813825a8ca8, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680922556095 2023-04-08 02:55:56,655 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 4369ebd17a6745c7963808cf038f470f, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680922556446 2023-04-08 02:55:56,664 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#C#compaction#486 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:56,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:55:56,681 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-08 02:55:56,681 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:55:56,681 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:56,682 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:55:56,682 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:56,682 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:55:56,682 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:56,704 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/2572fdd8c62e44e5808adb015ec632e7 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/2572fdd8c62e44e5808adb015ec632e7 2023-04-08 02:55:56,713 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/C of 61e4c342830c067a5389bf73aa5cd767 into 2572fdd8c62e44e5808adb015ec632e7(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:56,713 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:55:56,713 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/C, priority=12, startTime=1680922556498; duration=0sec 2023-04-08 02:55:56,713 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:56,713 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:55:56,714 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48234 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:55:56,714 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/B is initiating minor compaction (all files) 2023-04-08 02:55:56,714 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/B in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:56,714 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/44f1bb9bc5d749a5bbbe625f6b0d5ae9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/83c84fa075f943f3b7e6e6d9707b079d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/80d87c3dfd4c4b2ead0d5f078da49b8b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/eb4cbe428e0b4dd382eefc013ee7794e] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=47.1 K 2023-04-08 02:55:56,714 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 44f1bb9bc5d749a5bbbe625f6b0d5ae9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=120, earliestPutTs=1680922554890 2023-04-08 02:55:56,715 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 83c84fa075f943f3b7e6e6d9707b079d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=155, earliestPutTs=1680922554947 2023-04-08 02:55:56,715 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 80d87c3dfd4c4b2ead0d5f078da49b8b, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680922556095 2023-04-08 02:55:56,715 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting eb4cbe428e0b4dd382eefc013ee7794e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680922556446 2023-04-08 02:55:56,722 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:56,722 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922616717, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,723 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#B#compaction#488 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:56,729 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:56,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922616720, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:56,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922616721, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,730 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:56,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 84 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922616722, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,733 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/c16abcf0e47f4c568e7ff0df740a291b as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/c16abcf0e47f4c568e7ff0df740a291b 2023-04-08 02:55:56,739 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/B of 61e4c342830c067a5389bf73aa5cd767 into c16abcf0e47f4c568e7ff0df740a291b(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:56,740 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:55:56,740 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/B, priority=12, startTime=1680922556498; duration=0sec 2023-04-08 02:55:56,740 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:56,753 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:56,754 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:56,756 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:56,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:56,757 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:56,757 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:56,757 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:56,757 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:56,757 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:56,758 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,758 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:56,759 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:56,759 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:56,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:56,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:56,760 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:56,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:56,760 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:56,760 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,761 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,761 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:56,768 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:56,768 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:56,768 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:56,768 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:56,768 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:56,768 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:56,768 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:56,768 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:56,768 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:56,769 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:56,769 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:56,769 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:56,769 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:56,769 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,769 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:56,769 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:56,771 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,771 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:56,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:56,772 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,772 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:56,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:56,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:56,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:56,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:56,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:56,773 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@86f334e[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:56,773 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:56,773 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,776 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,776 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,776 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:56,776 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:56,776 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:56,776 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:56,778 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,778 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:56,778 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:56,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:56,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:56,778 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:56,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:56,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:56,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:56,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:56,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:56,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:56,781 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:56,781 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:56,781 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:56,781 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:56,781 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:56,781 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:56,782 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,782 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,782 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,782 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:56,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:56,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:56,783 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:56,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:56,786 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:56,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:56,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,790 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:56,790 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,790 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:56,790 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:56,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:56,790 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:56,790 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:56,790 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:56,790 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,790 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,790 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,790 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:56,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:56,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:56,790 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:56,790 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,791 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:56,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:56,828 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:56,828 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922616823, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,834 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:56,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922616831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:56,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922616831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,836 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:56,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922616831, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,891 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:56,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:56,894 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:56,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:56,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:56,895 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:56,895 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:56,895 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:56,895 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:56,896 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,896 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:56,898 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,898 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:56,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:56,898 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,898 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:56,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:56,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:56,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:56,899 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:56,899 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:56,899 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:56,899 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:56,899 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:56,899 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:56,908 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:56,908 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:56,908 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:56,908 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:56,908 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:56,908 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:56,908 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:56,908 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:56,908 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,908 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:56,908 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:56,910 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,910 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:56,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:56,911 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,911 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:56,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:56,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:56,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:56,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:56,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:56,912 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bd1aff[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:56,912 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:56,912 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,913 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,913 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:56,913 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:56,913 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:56,913 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,913 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:56,918 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:56,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:56,918 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:56,918 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:56,918 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:56,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:56,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:56,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:56,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:56,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:56,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:56,920 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:56,920 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:56,920 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:56,920 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:56,920 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:56,920 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:56,922 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,922 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,922 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:56,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:56,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:56,923 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:56,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:56,924 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,925 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:56,926 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:56,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:56,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:56,926 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:56,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:56,930 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,930 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:56,930 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:56,930 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:56,930 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,930 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:56,930 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:56,930 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:56,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:56,930 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:56,931 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:56,931 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:56,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:56,931 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:56,932 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/432cca971e7645b28af7d8df906a1cd2 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/432cca971e7645b28af7d8df906a1cd2 2023-04-08 02:55:56,936 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/A of 61e4c342830c067a5389bf73aa5cd767 into 432cca971e7645b28af7d8df906a1cd2(size=12.1 K), total size for store is 23.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:56,936 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:55:56,936 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/A, priority=13, startTime=1680922556498; duration=0sec 2023-04-08 02:55:56,936 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:57,031 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:57,032 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:57,034 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:57,034 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:57,034 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922617031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:57,035 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:57,035 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:57,035 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:57,035 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:57,035 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:57,035 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,035 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,037 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,037 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:57,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:57,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:57,037 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,037 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:57,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,037 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:57,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,038 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:57,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:57,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922617036, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,041 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:57,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922617037, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,046 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:57,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922617042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,048 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:57,048 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:57,048 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:57,048 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:57,048 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:57,051 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:57,052 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:57,052 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:57,052 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:57,052 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:57,052 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:57,052 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:57,052 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:57,052 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,052 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:57,052 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:57,057 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,057 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:57,057 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,057 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:57,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,058 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:57,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:57,059 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@296194df[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:57,059 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:57,059 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,061 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,061 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,061 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,061 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:57,061 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:57,061 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:57,061 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:57,063 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:57,063 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,063 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:57,063 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:57,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:57,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:57,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:57,066 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:57,066 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:57,066 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:57,066 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:57,066 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:57,066 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:57,068 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,068 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,068 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:57,068 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:57,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:57,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:57,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:57,069 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:57,070 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:57,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,071 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:57,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,075 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:57,075 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:57,075 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:57,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:57,076 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,076 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,076 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,076 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,076 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,076 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,076 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:57,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:57,076 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:57,077 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:57,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:57,077 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:57,077 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:57,077 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:57,106 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/3e653d853970411ea5720da3395d8a2d 2023-04-08 02:55:57,110 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:57,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922617105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,177 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:57,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:57,180 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:57,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:57,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:57,181 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:57,181 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:57,181 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:57,181 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:57,181 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,181 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,184 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:57,184 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:57,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,185 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:57,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,186 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:57,186 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:57,186 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:57,186 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:57,186 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:57,186 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:57,188 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:57,188 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,188 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:57,188 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:57,188 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,188 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:57,188 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:57,188 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:57,188 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:57,189 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:57,189 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:57,189 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,189 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:57,189 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:57,190 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,190 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,190 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:57,191 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:57,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:57,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:57,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:57,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56a6b677[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:57,193 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:57,193 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,194 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,194 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,194 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:57,194 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:57,194 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:57,194 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,195 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:57,197 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:57,197 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:57,197 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:57,197 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:57,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:57,199 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:57,199 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:57,200 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:57,200 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:57,200 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:57,200 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:57,201 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,201 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:57,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,201 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:57,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:57,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:57,202 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:57,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,203 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,207 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:57,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:57,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:57,208 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:57,208 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:57,208 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:57,208 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,208 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:57,208 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:57,208 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:57,208 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,208 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,208 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,208 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,208 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,208 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:57,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:57,308 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:57,309 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:57,310 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:57,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:57,324 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:57,324 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:57,324 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:57,324 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:57,324 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:57,325 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,325 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,327 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:57,327 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:57,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,327 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,327 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:57,328 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:57,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,328 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:57,328 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:57,328 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:57,328 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:57,328 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:57,328 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:57,329 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:57,329 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:57,330 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:57,330 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:57,331 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:57,331 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:57,331 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:57,331 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:57,331 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,331 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:57,331 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:57,333 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,333 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:57,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:57,334 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,334 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:57,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:57,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:57,335 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@27a21aae[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:57,335 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:57,335 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,336 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,337 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,337 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,337 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:57,337 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:57,337 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:57,337 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,337 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:57,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:57,338 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:57,338 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:57,338 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:57,338 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922617335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,338 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:57,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:57,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:57,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:57,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:57,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:57,341 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:57,341 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:57,341 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:57,341 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:57,341 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:57,341 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:57,346 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:57,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922617344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,346 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:57,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922617344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,347 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,347 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:57,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,347 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:57,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:57,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,348 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,348 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:57,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:57,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:57,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,349 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:57,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:57,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,351 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,351 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,351 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,351 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,351 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,351 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:57,351 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:57,351 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:57,351 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:57,351 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:57,352 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:57,352 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:57,351 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:57,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:57,352 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:57,352 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:57,352 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:57,352 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:57,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922617348, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,452 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:57,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:57,454 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:57,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:57,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:57,456 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:57,456 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:57,456 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:57,456 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:57,457 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,457 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,458 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:57,458 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,458 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:57,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:57,458 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,458 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:57,458 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,459 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:57,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,459 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:57,459 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:57,459 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:57,459 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:57,459 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:57,459 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:57,460 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:57,460 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:57,460 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:57,460 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:57,460 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:57,460 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:57,461 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:57,461 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:57,461 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,461 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:57,461 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:57,462 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,462 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:57,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:57,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:57,463 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,464 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:57,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:57,467 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d229647[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:57,467 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:57,467 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,472 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,473 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:57,473 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:57,473 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:57,473 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,473 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:57,474 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,474 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:57,474 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:57,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:57,474 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:57,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:57,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:57,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:57,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:57,479 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:57,480 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:57,480 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:57,480 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:57,480 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:57,480 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:57,482 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,482 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:57,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,482 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:57,482 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:57,482 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:57,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:57,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,484 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:57,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,490 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,490 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,490 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,490 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:57,490 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:57,490 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:57,490 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:57,490 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,490 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,490 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:57,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:57,490 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:57,490 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:57,490 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:57,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:57,542 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/b578871f0ed14c84bb778dd12f59a1b2 2023-04-08 02:55:57,590 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:57,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:57,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:57,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:57,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:57,595 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:57,595 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:57,595 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:57,595 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:57,596 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,596 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,598 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:57,598 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:57,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:57,599 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,599 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:57,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,599 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:57,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,599 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:57,600 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:57,600 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:57,600 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:57,600 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:57,600 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:57,616 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:57,616 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:57,619 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:57,619 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:57,619 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:57,619 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:57,619 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:57,619 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:57,619 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,619 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:57,620 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:57,621 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,621 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:57,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:57,622 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,622 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:57,622 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,623 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:57,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:57,623 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@68402450[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:57,623 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:57,623 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,624 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,625 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,625 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,625 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:57,625 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:57,625 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:57,625 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,625 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:57,627 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:57,627 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,627 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:57,627 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:57,627 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:57,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:57,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:57,629 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:57,629 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:57,629 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:57,629 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:57,629 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:57,629 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:57,636 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,637 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:57,637 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:57,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:57,637 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:57,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:57,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,638 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:57,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,644 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:57,644 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:57,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:57,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:57,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:57,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:57,646 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,646 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,646 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:57,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:57,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:57,646 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,647 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,647 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,647 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,647 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,647 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:57,647 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:57,747 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:57,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:57,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:57,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:57,752 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:57,752 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:57,752 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:57,752 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:57,752 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:57,753 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,753 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,755 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:57,756 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:57,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:57,756 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,756 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:57,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,757 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:57,757 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,757 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,757 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:57,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:57,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:57,760 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:57,760 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:57,760 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:57,764 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:57,764 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:57,764 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:57,764 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:57,764 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:57,764 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:57,764 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:57,764 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:57,764 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,764 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:57,764 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:57,766 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,766 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:57,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:57,766 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,766 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:57,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:57,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:57,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:57,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:57,768 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bb63221[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:57,768 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:57,768 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,770 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,770 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:57,770 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:57,770 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:57,770 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,770 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:57,771 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,771 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:57,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:57,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:57,772 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:57,772 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:57,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:57,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:57,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:57,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:57,776 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:57,776 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:57,777 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:57,777 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:57,777 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:57,777 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:57,778 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,778 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:57,778 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:57,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:57,779 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:57,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:57,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:57,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,779 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,779 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:57,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,780 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:57,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,786 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,786 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:57,786 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:57,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:57,786 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:57,787 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:57,786 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,787 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,786 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:57,786 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:57,787 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,787 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,787 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:57,787 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,787 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:57,787 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:57,787 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:57,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:57,842 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:57,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922617839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,851 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:57,851 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922617847, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,855 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:57,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922617849, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,857 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:57,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922617856, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,887 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:57,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:57,890 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:57,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:57,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:57,891 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:57,891 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:57,891 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:57,891 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:57,892 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,892 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,896 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:57,896 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:57,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:57,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,897 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:57,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,897 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:57,898 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,898 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:57,898 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:57,898 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:57,898 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:57,898 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:57,898 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,898 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:57,898 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:57,898 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:57,898 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:57,898 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:57,898 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:57,898 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:57,898 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:57,898 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:57,899 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,899 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:57,899 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:57,900 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,900 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:57,901 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,901 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:57,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:57,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:57,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3cf63e93[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:57,903 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:57,903 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,908 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,908 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,908 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:57,908 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:57,908 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:57,908 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:57,912 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,912 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:57,912 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:57,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:57,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:57,912 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:57,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:57,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:57,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:57,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:57,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:57,915 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:57,915 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:57,915 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:57,915 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:57,915 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:57,918 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,918 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:57,918 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:57,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:57,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:57,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:57,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:57,919 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:57,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,921 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:57,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,924 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:57,924 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:57,924 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:57,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:57,925 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:57,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:57,925 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,925 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,925 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:57,925 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:57,925 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,925 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:57,925 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:57,925 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:57,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:57,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:57,926 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:57,926 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:57,996 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=203 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/4f114db4b039495c87c31be92a6d7fd9 2023-04-08 02:55:58,000 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/3e653d853970411ea5720da3395d8a2d as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/3e653d853970411ea5720da3395d8a2d 2023-04-08 02:55:58,003 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/3e653d853970411ea5720da3395d8a2d, entries=200, sequenceid=203, filesize=14.1 K 2023-04-08 02:55:58,004 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/b578871f0ed14c84bb778dd12f59a1b2 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/b578871f0ed14c84bb778dd12f59a1b2 2023-04-08 02:55:58,016 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/b578871f0ed14c84bb778dd12f59a1b2, entries=150, sequenceid=203, filesize=11.8 K 2023-04-08 02:55:58,017 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/4f114db4b039495c87c31be92a6d7fd9 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/4f114db4b039495c87c31be92a6d7fd9 2023-04-08 02:55:58,021 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/4f114db4b039495c87c31be92a6d7fd9, entries=150, sequenceid=203, filesize=11.8 K 2023-04-08 02:55:58,022 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 61e4c342830c067a5389bf73aa5cd767 in 1341ms, sequenceid=203, compaction requested=true 2023-04-08 02:55:58,022 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:55:58,022 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:58,022 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:58,022 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-08 02:55:58,022 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:58,022 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:55:58,023 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-08 02:55:58,023 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-08 02:55:58,023 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. because compaction request was cancelled 2023-04-08 02:55:58,023 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-08 02:55:58,026 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:58,026 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 38840 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:55:58,027 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/A is initiating minor compaction (all files) 2023-04-08 02:55:58,027 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/A in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:58,027 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/432cca971e7645b28af7d8df906a1cd2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/4756e463510f42b0ab4c20bccb76d650, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/3e653d853970411ea5720da3395d8a2d] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=37.9 K 2023-04-08 02:55:58,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:58,028 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-08 02:55:58,028 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-08 02:55:58,028 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. because compaction request was cancelled 2023-04-08 02:55:58,029 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:58,030 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 432cca971e7645b28af7d8df906a1cd2, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=169, earliestPutTs=1680922556095 2023-04-08 02:55:58,030 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 4756e463510f42b0ab4c20bccb76d650, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680922556446 2023-04-08 02:55:58,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:58,030 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 3e653d853970411ea5720da3395d8a2d, keycount=200, bloomtype=ROW, size=14.1 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680922556557 2023-04-08 02:55:58,030 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:58,031 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:58,031 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:58,031 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:58,031 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:58,031 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:58,031 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:58,032 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:58,034 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:58,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:58,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:58,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:58,034 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:58,034 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:58,036 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:58,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:58,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:58,038 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:58,041 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:58,041 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:58,041 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:58,041 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:58,041 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:58,042 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#A#compaction#491 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:58,047 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:58,047 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:58,047 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:58,047 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:58,048 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-08 02:55:58,048 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:55:58,056 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:58,056 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:55:58,056 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:58,056 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:55:58,056 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:58,095 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/5b1f8d639ebd448ca52e52cb8c518da9 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/5b1f8d639ebd448ca52e52cb8c518da9 2023-04-08 02:55:58,101 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/A of 61e4c342830c067a5389bf73aa5cd767 into 5b1f8d639ebd448ca52e52cb8c518da9(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:58,101 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:55:58,101 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/A, priority=13, startTime=1680922558022; duration=0sec 2023-04-08 02:55:58,101 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:58,506 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=225 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/da00a91369824e3f912e2391b92877b6 2023-04-08 02:55:58,538 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=225 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/e13f05eb39cd443d92a2d1298680a410 2023-04-08 02:55:58,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:55:58,850 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:58,911 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:58,911 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 98 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922618870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:58,912 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:58,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922618907, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:58,916 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:58,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922618912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:58,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:58,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922618912, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:58,978 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=225 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/e8425a05f3d94f30a6af6b9abf0ccdc5 2023-04-08 02:55:58,983 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/da00a91369824e3f912e2391b92877b6 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/da00a91369824e3f912e2391b92877b6 2023-04-08 02:55:58,993 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/da00a91369824e3f912e2391b92877b6, entries=150, sequenceid=225, filesize=11.8 K 2023-04-08 02:55:58,994 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/e13f05eb39cd443d92a2d1298680a410 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/e13f05eb39cd443d92a2d1298680a410 2023-04-08 02:55:58,999 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/e13f05eb39cd443d92a2d1298680a410, entries=150, sequenceid=225, filesize=11.8 K 2023-04-08 02:55:59,000 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/e8425a05f3d94f30a6af6b9abf0ccdc5 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/e8425a05f3d94f30a6af6b9abf0ccdc5 2023-04-08 02:55:59,012 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/e8425a05f3d94f30a6af6b9abf0ccdc5, entries=150, sequenceid=225, filesize=11.8 K 2023-04-08 02:55:59,013 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 61e4c342830c067a5389bf73aa5cd767 in 966ms, sequenceid=225, compaction requested=true 2023-04-08 02:55:59,013 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:55:59,013 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:59,013 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:59,013 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:59,013 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,013 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:59,013 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:59,016 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,016 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:59,016 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:59,016 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,016 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:59,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:59,018 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:59,019 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@65bf17e4[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:59,019 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:59,019 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,021 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:55:59,021 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-08 02:55:59,022 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,022 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,022 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,022 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:59,022 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:59,022 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:59,022 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,022 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:59,022 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:55:59,023 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:59,023 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:55:59,023 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:59,023 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:55:59,023 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:59,024 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:59,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:59,025 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:59,025 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:59,025 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:59,025 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:59,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:59,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:59,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:59,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:59,029 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:59,029 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:59,029 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:59,029 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:59,029 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:59,031 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,031 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:59,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:59,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,031 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:59,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:59,034 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:59,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:59,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,034 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,034 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/2313e478712345f5a6a19734b8f63f34 2023-04-08 02:55:59,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:59,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,040 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,040 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:59,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,043 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:59,043 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:59,043 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:59,043 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,043 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:59,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:59,043 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,044 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,044 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,043 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:59,044 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:59,044 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,044 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,044 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:59,044 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:59,044 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,045 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:59,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:59,064 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:59,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922619058, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,065 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:59,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922619063, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:59,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922619064, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:59,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922619065, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,135 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:59,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922619130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,141 DEBUG [Thread-3372] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4189 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=2, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:55:59,144 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:59,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:59,147 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:59,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:59,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:59,148 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:59,148 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:59,148 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:59,148 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:59,148 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,149 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,150 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:59,150 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,150 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:59,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:59,150 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,150 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:59,150 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,151 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:59,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,151 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:59,152 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:59,152 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:59,152 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:59,152 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:59,152 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:59,152 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:59,152 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:59,152 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:59,152 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:59,152 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:59,152 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:59,152 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:59,152 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:59,152 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,152 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:59,152 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:59,155 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,155 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:59,155 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:59,155 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,155 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:59,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:59,157 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cb4dda7[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:59,157 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:59,157 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,159 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,159 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,159 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,159 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:59,159 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:59,159 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:59,160 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,160 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:59,168 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,169 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:59,169 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:59,169 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:59,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:59,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:59,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922619165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,170 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:59,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,170 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922619166, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:59,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,171 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:59,172 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:59,172 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:59,172 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:59,172 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:59,172 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:59,172 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:59,175 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,176 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:59,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,176 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:59,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:59,176 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,177 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:59,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:59,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:59,177 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,177 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,178 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:59,179 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922619174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,179 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:59,179 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922619174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:59,185 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:59,185 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:59,185 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:59,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:59,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:59,185 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:59,185 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,185 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,185 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,185 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,185 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,185 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,185 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:59,185 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:59,185 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:59,185 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,186 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:59,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:59,186 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:59,285 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:59,286 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:59,288 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:59,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:59,289 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:59,289 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:59,289 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:59,289 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:59,289 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:59,289 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,289 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,291 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:59,291 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,291 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:59,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:59,291 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,291 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:59,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,292 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:59,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,292 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:59,292 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:59,292 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:59,293 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:59,293 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:59,293 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:59,296 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:59,296 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:59,296 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:59,296 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:59,296 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:59,296 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:59,296 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:59,296 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:59,296 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,296 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:59,296 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:59,298 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,298 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:59,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:59,298 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:59,298 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:59,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:59,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:59,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:59,300 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@36a9c12a[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:59,300 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:59,300 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,303 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,303 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:59,303 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,303 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:59,303 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:59,303 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:59,304 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,304 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:59,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:59,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:59,304 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:59,305 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:59,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:59,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:59,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:59,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:59,307 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:59,307 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:59,308 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:59,308 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:59,308 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:59,308 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:59,309 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,309 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:59,309 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:59,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:59,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:59,311 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:59,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,313 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,320 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:59,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,326 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:59,326 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:59,326 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:59,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:59,327 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:59,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:59,328 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,328 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,328 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,328 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,328 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,328 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,328 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,328 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:59,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:59,328 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:59,329 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:59,329 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:59,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:59,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922619372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,376 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:59,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922619372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:59,385 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:59,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922619380, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922619381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,430 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:59,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:59,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:59,432 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:59,433 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:59,433 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:59,433 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:59,433 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:59,433 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:59,433 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,433 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,436 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,436 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:59,436 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:59,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:59,436 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,436 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:59,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,436 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:59,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,437 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:59,437 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:59,437 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:59,437 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:59,437 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:59,437 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:59,437 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:59,437 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:59,438 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:59,438 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:59,438 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:59,438 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:59,438 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:59,438 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:59,438 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,438 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:59,438 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:59,439 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,439 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:59,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:59,440 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,440 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:59,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:59,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:59,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,441 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:59,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:59,441 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3ba663ca[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:59,441 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:59,441 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,442 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,442 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:59,442 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:59,442 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:59,442 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,442 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:59,443 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:59,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:59,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:59,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:59,444 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:59,445 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:59,445 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:59,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,445 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:59,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:59,446 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:59,446 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:59,447 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:59,447 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:59,447 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:59,447 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:59,450 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,450 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:59,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,450 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:59,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:59,450 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:59,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:59,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:59,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,451 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:59,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,453 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:59,453 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,455 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,455 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,455 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:59,455 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,455 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:59,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:59,455 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:59,455 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:59,455 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:59,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:59,455 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:59,455 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,456 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,456 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,456 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:59,456 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,456 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:59,456 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:59,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:59,461 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/1bf655056dfc44318d6b48e356abecea 2023-04-08 02:55:59,556 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:59,556 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:59,558 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:59,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:59,559 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:59,559 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:59,559 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:59,559 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:59,559 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:59,559 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,559 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,561 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:59,561 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,561 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:59,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:59,561 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,561 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:59,561 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,561 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:59,562 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,562 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:59,562 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:59,562 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:59,562 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:59,562 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:59,562 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:59,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:59,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:59,564 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:59,564 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:59,564 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:59,564 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:59,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:59,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:59,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:59,564 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:59,566 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,566 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:59,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:59,566 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:59,566 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:59,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:59,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:59,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:59,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@263b536d[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:59,567 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:59,567 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,568 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,568 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,568 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,568 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:59,568 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:59,568 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:59,568 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,569 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:59,571 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,571 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:59,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:59,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:59,571 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:59,571 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:59,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:59,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:59,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:59,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:59,573 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:59,573 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:59,573 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:59,573 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:59,573 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:59,573 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:59,575 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,575 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,575 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:59,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:59,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:59,575 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:59,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:59,575 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:59,575 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:59,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,577 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:59,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,579 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,579 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,579 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,579 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,579 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,579 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:59,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:59,579 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:59,579 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:55:59,579 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,579 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:59,580 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:59,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:59,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:59,580 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:59,580 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:59,580 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:59,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:59,680 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:59,681 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:59,682 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:59,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:59,683 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:59,683 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:59,683 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:59,684 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:59,684 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:59,684 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,684 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,684 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:59,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922619677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,684 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:59,685 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922619677, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,686 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:59,686 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:59,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:59,686 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,686 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:59,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,692 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:59,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,693 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:59,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:59,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922619687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,694 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:59,694 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:55:59,694 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:59,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922619694, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,694 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:59,694 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:59,694 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:59,696 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:59,696 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:59,700 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:59,700 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:59,700 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:59,700 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:59,700 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:59,701 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:59,701 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,701 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:59,701 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:59,715 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,715 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:59,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:59,715 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,715 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:59,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:59,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:59,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:59,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:59,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@cc8fbe8[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:59,717 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:59,717 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,722 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,722 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,722 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:59,722 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:59,722 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:59,722 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,723 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:59,724 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:59,724 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,724 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:59,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:59,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:59,724 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:59,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:59,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:59,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:59,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:59,733 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:59,733 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:59,733 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:59,733 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:59,733 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:59,733 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:59,738 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,738 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:59,738 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,738 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:59,739 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:59,740 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:59,741 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:59,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,743 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,744 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,753 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:59,753 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:59,753 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,753 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,753 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,753 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,753 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,753 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,754 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:59,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:59,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:59,754 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:59,754 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:59,754 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:59,754 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:59,754 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:59,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:59,854 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:59,855 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:59,856 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:59,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:59,857 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:59,857 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:59,857 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:59,857 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:59,857 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:59,858 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,858 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,859 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:59,859 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,859 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:59,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:59,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,860 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,860 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:59,860 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:59,861 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,861 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:59,861 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:59,861 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:59,861 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:59,861 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:59,861 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:59,861 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:59,861 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:59,861 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:59,861 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:59,862 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:55:59,862 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:59,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:55:59,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:55:59,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:55:59,862 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:55:59,863 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,863 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,863 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:59,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:59,864 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:59,864 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,865 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:55:59,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:59,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,865 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:59,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:55:59,867 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7410cbca[Count = 0] remaining members to acquire global barrier 2023-04-08 02:55:59,867 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:55:59,867 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,869 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,869 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:55:59,869 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:55:59,869 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:55:59,869 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,869 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:55:59,870 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:55:59,870 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:59,870 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,870 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:55:59,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:59,870 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:59,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:59,871 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:59,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:59,873 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,874 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=241 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/51e23a1b7a984d848a652342d252efa7 2023-04-08 02:55:59,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:55:59,875 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:55:59,875 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:55:59,876 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:55:59,876 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:55:59,876 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:55:59,876 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:55:59,878 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,878 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,878 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:59,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:55:59,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:55:59,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:55:59,879 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:59,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:59,879 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/2313e478712345f5a6a19734b8f63f34 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/2313e478712345f5a6a19734b8f63f34 2023-04-08 02:55:59,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,880 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:55:59,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,881 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,882 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:55:59,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:55:59,883 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/2313e478712345f5a6a19734b8f63f34, entries=150, sequenceid=241, filesize=11.8 K 2023-04-08 02:55:59,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,884 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/1bf655056dfc44318d6b48e356abecea as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/1bf655056dfc44318d6b48e356abecea 2023-04-08 02:55:59,886 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:59,886 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:55:59,886 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:59,886 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:55:59,886 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:55:59,886 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:55:59,886 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:55:59,886 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:55:59,886 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:59,886 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,887 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,887 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,887 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:55:59,887 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:55:59,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:55:59,888 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/1bf655056dfc44318d6b48e356abecea, entries=150, sequenceid=241, filesize=11.8 K 2023-04-08 02:55:59,889 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/51e23a1b7a984d848a652342d252efa7 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/51e23a1b7a984d848a652342d252efa7 2023-04-08 02:55:59,894 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/51e23a1b7a984d848a652342d252efa7, entries=150, sequenceid=241, filesize=11.8 K 2023-04-08 02:55:59,894 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 61e4c342830c067a5389bf73aa5cd767 in 873ms, sequenceid=241, compaction requested=true 2023-04-08 02:55:59,895 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:55:59,895 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:55:59,895 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:59,895 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:59,895 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:55:59,895 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:59,896 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36552 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:55:59,896 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/A is initiating minor compaction (all files) 2023-04-08 02:55:59,896 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/A in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:59,896 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/5b1f8d639ebd448ca52e52cb8c518da9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/da00a91369824e3f912e2391b92877b6, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/2313e478712345f5a6a19734b8f63f34] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=35.7 K 2023-04-08 02:55:59,897 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 5b1f8d639ebd448ca52e52cb8c518da9, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680922556557 2023-04-08 02:55:59,897 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48520 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:55:59,897 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/B is initiating minor compaction (all files) 2023-04-08 02:55:59,897 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/B in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:59,897 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting da00a91369824e3f912e2391b92877b6, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1680922556719 2023-04-08 02:55:59,897 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/c16abcf0e47f4c568e7ff0df740a291b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/b578871f0ed14c84bb778dd12f59a1b2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/e13f05eb39cd443d92a2d1298680a410, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/1bf655056dfc44318d6b48e356abecea] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=47.4 K 2023-04-08 02:55:59,897 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 2313e478712345f5a6a19734b8f63f34, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1680922558910 2023-04-08 02:55:59,897 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting c16abcf0e47f4c568e7ff0df740a291b, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680922556446 2023-04-08 02:55:59,898 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting b578871f0ed14c84bb778dd12f59a1b2, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680922556557 2023-04-08 02:55:59,898 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting e13f05eb39cd443d92a2d1298680a410, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1680922556719 2023-04-08 02:55:59,898 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 1bf655056dfc44318d6b48e356abecea, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1680922558910 2023-04-08 02:55:59,905 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#B#compaction#498 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:59,913 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#A#compaction#499 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:59,938 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/7197bf88c4854ced8f6398cb10a5c21c as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/7197bf88c4854ced8f6398cb10a5c21c 2023-04-08 02:55:59,946 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/B of 61e4c342830c067a5389bf73aa5cd767 into 7197bf88c4854ced8f6398cb10a5c21c(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:59,946 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:55:59,946 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/B, priority=12, startTime=1680922559895; duration=0sec 2023-04-08 02:55:59,946 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:55:59,946 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:55:59,947 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48520 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:55:59,947 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/C is initiating minor compaction (all files) 2023-04-08 02:55:59,947 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/C in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:59,947 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/2572fdd8c62e44e5808adb015ec632e7, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/4f114db4b039495c87c31be92a6d7fd9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/e8425a05f3d94f30a6af6b9abf0ccdc5, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/51e23a1b7a984d848a652342d252efa7] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=47.4 K 2023-04-08 02:55:59,948 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 2572fdd8c62e44e5808adb015ec632e7, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=189, earliestPutTs=1680922556446 2023-04-08 02:55:59,948 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 4f114db4b039495c87c31be92a6d7fd9, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=203, earliestPutTs=1680922556557 2023-04-08 02:55:59,949 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting e8425a05f3d94f30a6af6b9abf0ccdc5, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=225, earliestPutTs=1680922556719 2023-04-08 02:55:59,949 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 51e23a1b7a984d848a652342d252efa7, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1680922558910 2023-04-08 02:55:59,955 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#C#compaction#500 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:55:59,973 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/464b24b807634d5ca8c8f2a0ffd0d9a7 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/464b24b807634d5ca8c8f2a0ffd0d9a7 2023-04-08 02:55:59,978 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/C of 61e4c342830c067a5389bf73aa5cd767 into 464b24b807634d5ca8c8f2a0ffd0d9a7(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:55:59,978 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:55:59,978 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/C, priority=12, startTime=1680922559895; duration=0sec 2023-04-08 02:55:59,978 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:55:59,987 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:55:59,988 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:55:59,989 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:55:59,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:55:59,990 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:55:59,990 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:55:59,990 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:59,990 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:55:59,990 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:55:59,991 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,991 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,994 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:55:59,994 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:55:59,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:55:59,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,994 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,995 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:55:59,995 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:55:59,995 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:55:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:55:59,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:55:59,995 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:55:59,996 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:55:59,996 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:55:59,996 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:55:59,996 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:55:59,996 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:55:59,996 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:55:59,996 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:55:59,996 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:55:59,996 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:55:59,996 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-08 02:55:59,996 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:55:59,997 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:59,997 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:55:59,997 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:55:59,997 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:55:59,997 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:00,187 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:56:00,188 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:00,203 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:00,203 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922620200, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:00,204 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:00,204 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922620202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:00,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:00,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 114 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922620203, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:00,207 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:00,207 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922620204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:00,305 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:00,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922620304, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:00,305 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:00,305 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922620305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:00,309 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:00,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 116 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922620308, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:00,339 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/440c79ecf73d46449bfbee86bdd9d776 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/440c79ecf73d46449bfbee86bdd9d776 2023-04-08 02:56:00,343 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/A of 61e4c342830c067a5389bf73aa5cd767 into 440c79ecf73d46449bfbee86bdd9d776(size=12.3 K), total size for store is 12.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:00,343 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:00,343 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/A, priority=13, startTime=1680922559895; duration=0sec 2023-04-08 02:56:00,343 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:00,410 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/aff25e5b612849c3abc53f12b9889f88 2023-04-08 02:56:00,510 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:00,511 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:00,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922620507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:00,512 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922620507, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:00,513 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:00,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 118 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922620511, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:00,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:00,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922620813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:00,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:00,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922620813, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:00,818 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:00,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 120 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922620815, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:00,819 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/b23c166b2a644a79926a3193e9bebd12 2023-04-08 02:56:01,218 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:01,218 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922621214, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,246 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=261 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/e8f73f8e37444d33aa79256b95582740 2023-04-08 02:56:01,255 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/aff25e5b612849c3abc53f12b9889f88 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/aff25e5b612849c3abc53f12b9889f88 2023-04-08 02:56:01,259 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/aff25e5b612849c3abc53f12b9889f88, entries=150, sequenceid=261, filesize=11.9 K 2023-04-08 02:56:01,260 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/b23c166b2a644a79926a3193e9bebd12 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/b23c166b2a644a79926a3193e9bebd12 2023-04-08 02:56:01,280 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/b23c166b2a644a79926a3193e9bebd12, entries=150, sequenceid=261, filesize=11.9 K 2023-04-08 02:56:01,281 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/e8f73f8e37444d33aa79256b95582740 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/e8f73f8e37444d33aa79256b95582740 2023-04-08 02:56:01,285 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/e8f73f8e37444d33aa79256b95582740, entries=150, sequenceid=261, filesize=11.9 K 2023-04-08 02:56:01,286 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 61e4c342830c067a5389bf73aa5cd767 in 1290ms, sequenceid=261, compaction requested=false 2023-04-08 02:56:01,286 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:01,286 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:01,286 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:01,286 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:01,286 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,290 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:01,290 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:01,292 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,292 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:01,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:01,293 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,293 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:01,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:01,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:01,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:01,293 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,294 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:01,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:01,294 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7ca04eb1[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:01,294 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:01,294 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,295 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,296 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:01,296 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,296 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:01,296 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:01,296 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:01,297 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,297 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:01,297 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:01,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:01,297 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:01,297 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:01,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:01,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:01,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:01,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:01,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:01,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:01,299 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:01,299 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:01,299 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:01,299 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:01,299 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:01,299 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:01,302 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,302 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,302 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:01,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:01,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:01,302 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,303 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:01,303 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:01,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:01,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:01,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:01,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:01,304 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:01,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:01,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,308 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,308 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:01,308 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:01,308 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:01,308 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:01,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:01,309 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:01,309 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:01,309 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:01,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:01,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:01,309 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:01,309 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:01,308 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,309 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,309 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,309 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,309 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:01,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:01,325 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:56:01,325 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-08 02:56:01,325 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:56:01,325 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:01,325 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:56:01,325 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:01,325 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:56:01,325 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:01,338 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/0adf181db4ed475b9b1dc2fa25d1d2d5 2023-04-08 02:56:01,352 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/8c895cc7bba349e98cd2f2013a82cfd6 2023-04-08 02:56:01,356 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:01,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50250 deadline: 1680922621349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:01,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50232 deadline: 1680922621352, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,361 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:01,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50252 deadline: 1680922621353, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,409 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:01,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:01,411 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:01,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:01,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:01,412 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:01,413 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:01,413 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:01,413 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:01,414 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,414 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:01,416 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,416 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:01,416 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:01,416 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:01,416 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,416 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:01,418 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:01,419 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:01,419 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:01,419 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:01,419 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:01,419 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:01,419 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:01,419 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:01,420 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:01,420 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:01,420 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:01,420 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:01,420 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:01,420 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:01,420 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:01,420 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:01,420 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:01,420 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,420 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:01,420 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:01,422 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,422 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:01,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:01,424 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,424 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:01,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:01,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:01,424 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:01,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:01,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:01,425 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7cda73[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:01,425 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:01,425 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,430 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,430 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,431 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:01,431 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,431 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:01,431 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:01,431 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:01,432 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,432 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:01,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:01,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:01,432 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:01,433 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:01,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:01,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:01,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:01,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:01,434 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:01,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:01,435 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:01,435 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:01,435 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:01,435 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:01,435 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:01,435 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:01,437 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,437 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:01,437 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,437 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:01,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:01,438 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:01,438 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:01,439 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,439 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:01,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:01,440 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,442 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,442 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:01,442 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:01,442 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:01,442 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:01,442 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:01,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:01,442 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:01,442 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:01,442 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:01,442 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:01,443 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,443 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,443 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,443 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,443 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:01,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:01,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:01,460 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:01,460 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922621458, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:01,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922621462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,465 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:01,465 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922621462, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,543 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:01,543 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:01,545 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:01,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:01,546 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:01,546 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:01,546 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:01,546 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:01,546 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:01,546 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,546 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:01,548 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,548 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:01,548 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:01,548 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:01,549 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:01,549 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,549 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:01,549 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:01,550 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:01,550 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:01,550 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:01,550 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:01,550 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:01,550 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:01,550 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:01,550 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:01,550 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:01,551 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:01,551 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:01,551 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:01,551 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:01,551 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:01,551 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:01,551 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,551 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:01,551 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:01,552 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,552 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:01,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:01,553 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,553 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:01,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:01,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:01,553 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:01,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:01,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:01,554 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b20762f[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:01,554 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:01,554 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,556 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,556 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,556 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,556 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,556 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:01,556 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:01,556 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:01,556 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:01,558 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,558 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:01,558 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:01,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:01,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:01,558 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:01,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:01,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:01,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:01,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:01,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:01,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:01,560 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:01,560 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:01,560 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:01,561 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:01,561 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:01,561 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:01,563 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,563 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,563 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,563 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:01,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:01,563 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:01,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:01,564 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:01,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:01,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:01,564 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:01,565 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:01,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:01,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:01,566 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,569 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,569 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:01,569 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:01,569 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:01,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:01,569 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:01,569 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,569 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,569 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,569 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,569 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:01,569 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:01,576 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:01,577 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:01,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:01,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:01,577 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:01,577 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:01,665 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:01,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922621662, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,668 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:01,668 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922621666, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,670 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:01,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:01,672 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:01,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:01,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:01,673 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:01,673 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:01,673 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:01,673 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:01,673 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:01,673 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,674 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:01,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922621667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,675 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:01,675 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,675 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:01,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:01,675 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,675 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:01,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:01,676 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:01,676 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:01,676 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:01,676 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:01,676 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:01,676 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:01,676 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:01,676 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:01,677 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:01,677 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:01,677 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:01,677 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:01,677 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:01,677 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:01,677 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:01,677 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:01,677 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,677 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:01,677 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:01,678 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,678 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:01,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:01,679 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,679 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:01,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:01,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:01,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@399caa1a[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:01,681 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:01,681 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,683 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,683 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,683 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:01,683 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:01,683 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:01,684 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:01,684 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,684 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:01,685 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:01,685 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:01,685 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:01,685 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:01,685 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:01,685 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:01,685 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:01,685 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:01,686 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,686 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:01,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:01,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,686 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:01,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:01,687 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,687 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:01,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:01,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:01,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,687 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:01,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:01,688 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:01,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:01,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,690 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,690 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:01,690 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:01,690 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:01,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:01,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:01,690 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:01,690 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:01,690 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:01,690 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:01,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:01,690 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:01,690 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:01,690 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,691 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,691 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:01,691 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,691 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:01,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:01,765 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=278 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/c2cf393e27744e6ea53339de02dad836 2023-04-08 02:56:01,769 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/0adf181db4ed475b9b1dc2fa25d1d2d5 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/0adf181db4ed475b9b1dc2fa25d1d2d5 2023-04-08 02:56:01,772 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/0adf181db4ed475b9b1dc2fa25d1d2d5, entries=200, sequenceid=278, filesize=14.3 K 2023-04-08 02:56:01,773 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/8c895cc7bba349e98cd2f2013a82cfd6 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/8c895cc7bba349e98cd2f2013a82cfd6 2023-04-08 02:56:01,776 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/8c895cc7bba349e98cd2f2013a82cfd6, entries=150, sequenceid=278, filesize=11.9 K 2023-04-08 02:56:01,777 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/c2cf393e27744e6ea53339de02dad836 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/c2cf393e27744e6ea53339de02dad836 2023-04-08 02:56:01,780 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/c2cf393e27744e6ea53339de02dad836, entries=150, sequenceid=278, filesize=11.9 K 2023-04-08 02:56:01,781 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=87.22 KB/89310 for 61e4c342830c067a5389bf73aa5cd767 in 456ms, sequenceid=278, compaction requested=true 2023-04-08 02:56:01,781 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:01,781 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:01,781 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:56:01,781 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:56:01,781 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:01,781 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:01,782 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39344 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:56:01,782 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/A is initiating minor compaction (all files) 2023-04-08 02:56:01,782 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/A in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:01,782 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/440c79ecf73d46449bfbee86bdd9d776, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/aff25e5b612849c3abc53f12b9889f88, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/0adf181db4ed475b9b1dc2fa25d1d2d5] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=38.4 K 2023-04-08 02:56:01,782 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36870 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:56:01,782 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/B is initiating minor compaction (all files) 2023-04-08 02:56:01,782 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/B in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:01,782 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/7197bf88c4854ced8f6398cb10a5c21c, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/b23c166b2a644a79926a3193e9bebd12, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/8c895cc7bba349e98cd2f2013a82cfd6] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=36.0 K 2023-04-08 02:56:01,783 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 440c79ecf73d46449bfbee86bdd9d776, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1680922558910 2023-04-08 02:56:01,783 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 7197bf88c4854ced8f6398cb10a5c21c, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1680922558910 2023-04-08 02:56:01,783 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting aff25e5b612849c3abc53f12b9889f88, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1680922559054 2023-04-08 02:56:01,783 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting b23c166b2a644a79926a3193e9bebd12, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1680922559054 2023-04-08 02:56:01,783 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 0adf181db4ed475b9b1dc2fa25d1d2d5, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1680922560199 2023-04-08 02:56:01,784 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 8c895cc7bba349e98cd2f2013a82cfd6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1680922560199 2023-04-08 02:56:01,789 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#A#compaction#507 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:01,790 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#B#compaction#508 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:01,791 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:01,791 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:01,793 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:01,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:01,804 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:01,804 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:01,804 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:01,804 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:01,804 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:01,804 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,805 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:01,807 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:01,807 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:01,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:01,807 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:01,807 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:01,807 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:01,808 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:01,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:01,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:01,808 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:01,808 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:01,808 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:01,808 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:01,808 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:01,808 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:01,809 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:01,809 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:01,809 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:01,809 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:01,809 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=87.22 KB heapSize=259.73 KB 2023-04-08 02:56:01,809 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:56:01,809 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:01,810 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:56:01,810 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:01,810 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:56:01,810 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:01,810 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/934fcc2b5702463a99bb3e32425a2741 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/934fcc2b5702463a99bb3e32425a2741 2023-04-08 02:56:01,816 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/A of 61e4c342830c067a5389bf73aa5cd767 into 934fcc2b5702463a99bb3e32425a2741(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:01,817 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:01,817 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/A, priority=13, startTime=1680922561781; duration=0sec 2023-04-08 02:56:01,817 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:01,817 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:56:01,818 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 36870 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:56:01,819 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/C is initiating minor compaction (all files) 2023-04-08 02:56:01,819 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/C in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:01,819 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/712f7aa9f2db45fd9eccdc45367de9f7 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/712f7aa9f2db45fd9eccdc45367de9f7 2023-04-08 02:56:01,819 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/464b24b807634d5ca8c8f2a0ffd0d9a7, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/e8f73f8e37444d33aa79256b95582740, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/c2cf393e27744e6ea53339de02dad836] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=36.0 K 2023-04-08 02:56:01,819 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 464b24b807634d5ca8c8f2a0ffd0d9a7, keycount=150, bloomtype=ROW, size=12.3 K, encoding=NONE, compression=NONE, seqNum=241, earliestPutTs=1680922558910 2023-04-08 02:56:01,820 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting e8f73f8e37444d33aa79256b95582740, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=261, earliestPutTs=1680922559054 2023-04-08 02:56:01,820 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting c2cf393e27744e6ea53339de02dad836, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1680922560199 2023-04-08 02:56:01,826 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/B of 61e4c342830c067a5389bf73aa5cd767 into 712f7aa9f2db45fd9eccdc45367de9f7(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:01,826 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:01,826 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/B, priority=13, startTime=1680922561781; duration=0sec 2023-04-08 02:56:01,826 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:01,830 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#C#compaction#510 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:01,840 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/05fe345c4b3e49bfa7160bd3e07d6e11 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/05fe345c4b3e49bfa7160bd3e07d6e11 2023-04-08 02:56:01,848 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/C of 61e4c342830c067a5389bf73aa5cd767 into 05fe345c4b3e49bfa7160bd3e07d6e11(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:01,848 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:01,848 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/C, priority=13, startTime=1680922561781; duration=0sec 2023-04-08 02:56:01,848 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:01,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:56:01,970 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:02,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:02,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922622001, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,007 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:02,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922622006, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,010 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:02,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922622007, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,109 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:02,109 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922622108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:02,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922622108, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,113 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:02,114 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922622111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,232 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/ac64aff2506a49ff9352fabcbf3cf464 2023-04-08 02:56:02,253 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/f5f5aa780fee4142b149509eabbaa3a6 2023-04-08 02:56:02,313 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:02,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922622310, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,316 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:02,316 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922622313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,319 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:02,320 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922622314, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,618 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:02,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922622615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,623 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:02,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922622619, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,625 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:02,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922622622, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,670 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.07 KB at sequenceid=294 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/299b0b60150f41e69e44767b98320346 2023-04-08 02:56:02,676 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/ac64aff2506a49ff9352fabcbf3cf464 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/ac64aff2506a49ff9352fabcbf3cf464 2023-04-08 02:56:02,680 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/ac64aff2506a49ff9352fabcbf3cf464, entries=150, sequenceid=294, filesize=11.9 K 2023-04-08 02:56:02,680 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/f5f5aa780fee4142b149509eabbaa3a6 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/f5f5aa780fee4142b149509eabbaa3a6 2023-04-08 02:56:02,684 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/f5f5aa780fee4142b149509eabbaa3a6, entries=150, sequenceid=294, filesize=11.9 K 2023-04-08 02:56:02,684 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/299b0b60150f41e69e44767b98320346 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/299b0b60150f41e69e44767b98320346 2023-04-08 02:56:02,687 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/299b0b60150f41e69e44767b98320346, entries=150, sequenceid=294, filesize=11.9 K 2023-04-08 02:56:02,688 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~87.22 KB/89310, heapSize ~259.69 KB/265920, currentSize=93.93 KB/96180 for 61e4c342830c067a5389bf73aa5cd767 in 879ms, sequenceid=294, compaction requested=false 2023-04-08 02:56:02,688 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:02,688 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:02,688 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:02,688 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:02,688 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,688 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:02,689 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:02,690 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,690 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:02,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:02,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:02,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:02,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:02,692 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:02,692 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:02,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:02,692 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4737ae5e[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:02,692 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:02,693 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:02,694 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:02,694 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:02,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:02,694 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:02,694 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:02,694 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,694 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:02,694 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:02,696 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,696 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:02,696 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:02,696 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:02,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:02,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:02,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:02,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:02,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:02,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,701 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:02,701 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:02,701 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:02,701 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:02,701 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:02,701 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:02,701 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:02,708 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:02,708 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:02,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:02,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:02,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:02,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:02,708 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:02,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:02,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:02,709 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:02,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:02,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:02,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:02,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:02,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:02,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:02,712 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:02,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,716 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:02,716 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,716 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:02,717 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:02,716 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:02,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:02,716 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:02,716 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:02,717 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:02,717 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:02,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:02,717 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:02,717 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,717 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:02,717 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:02,717 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:02,717 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:02,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:02,817 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:02,818 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:02,820 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:02,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:02,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:02,821 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:02,821 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:02,821 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:02,821 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:02,823 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:02,823 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:02,824 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,824 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:02,824 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:02,824 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:02,825 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,825 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:02,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:02,825 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:02,825 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:02,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:02,826 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:02,826 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:02,826 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:02,826 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:02,826 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:02,826 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:02,826 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:02,826 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:02,826 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:02,826 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:02,827 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-08 02:56:02,827 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:56:02,827 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:02,827 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:56:02,827 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:02,827 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:56:02,827 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:02,859 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/1ab86acd67e3408f868aed861ed47827 2023-04-08 02:56:02,878 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/fa1e0045333b48a9948c5a7f55e973b4 2023-04-08 02:56:02,927 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=314 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/8a1ce1acaec14d8cb7308cb4462fb534 2023-04-08 02:56:02,932 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/1ab86acd67e3408f868aed861ed47827 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/1ab86acd67e3408f868aed861ed47827 2023-04-08 02:56:02,936 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/1ab86acd67e3408f868aed861ed47827, entries=150, sequenceid=314, filesize=11.9 K 2023-04-08 02:56:02,936 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/fa1e0045333b48a9948c5a7f55e973b4 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/fa1e0045333b48a9948c5a7f55e973b4 2023-04-08 02:56:02,943 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/fa1e0045333b48a9948c5a7f55e973b4, entries=150, sequenceid=314, filesize=11.9 K 2023-04-08 02:56:02,944 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/8a1ce1acaec14d8cb7308cb4462fb534 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/8a1ce1acaec14d8cb7308cb4462fb534 2023-04-08 02:56:02,948 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/8a1ce1acaec14d8cb7308cb4462fb534, entries=150, sequenceid=314, filesize=11.9 K 2023-04-08 02:56:02,949 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 61e4c342830c067a5389bf73aa5cd767 in 122ms, sequenceid=314, compaction requested=true 2023-04-08 02:56:02,949 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:02,949 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:02,949 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:02,949 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:02,949 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,949 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:02,949 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:02,952 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,952 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:02,952 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:02,952 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:02,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:02,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:02,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:02,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:02,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:02,954 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@217e5b89[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:02,954 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:02,954 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:02,956 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:02,956 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:02,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:02,956 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:02,956 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:02,956 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,956 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:02,956 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:02,958 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,958 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:02,958 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:02,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:02,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:02,958 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:02,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:02,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:02,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:02,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:02,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:02,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:02,960 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:02,960 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:02,960 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:02,960 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:02,960 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:02,960 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:02,962 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:02,962 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:02,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:02,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:02,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:02,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:02,962 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:02,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:02,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:02,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:02,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:02,963 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:02,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:02,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,964 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:02,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:02,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,967 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,967 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:02,967 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:02,967 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:02,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:02,967 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:02,967 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:02,967 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:02,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:02,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:02,967 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:02,967 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:02,967 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:02,967 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:02,967 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:02,967 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:02,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:02,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:03,067 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:03,068 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:03,070 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:03,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:03,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:03,072 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:03,072 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:03,072 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:03,072 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:03,072 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,072 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,074 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:03,074 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,074 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:03,074 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:03,075 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,075 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:03,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,075 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:03,075 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,075 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:03,076 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:03,076 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:03,076 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:03,076 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:03,076 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:03,077 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:03,077 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:03,077 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:03,077 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:03,077 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:03,077 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:03,077 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:03,078 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:03,078 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,078 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:03,078 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:03,079 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,079 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:03,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:03,079 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,079 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:03,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:03,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:03,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:03,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:03,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@69a7c087[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:03,086 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:03,087 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,088 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,088 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:03,088 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:03,088 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:03,088 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,088 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:03,092 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,092 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:03,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:03,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:03,092 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:03,092 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:03,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:03,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:03,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:03,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:03,094 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:03,094 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:03,094 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:03,094 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:03,094 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:03,094 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:03,097 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,097 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:03,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:03,097 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,097 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,097 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:03,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:03,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:03,098 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:03,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:03,101 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:03,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,104 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:03,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:03,104 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:03,104 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,104 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,104 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,104 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,104 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,104 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,104 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,104 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:03,104 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:03,104 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:03,104 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:03,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:03,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:03,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:03,105 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:03,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:03,134 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:56:03,135 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-08 02:56:03,135 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:56:03,135 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:03,135 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:56:03,135 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:03,135 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:56:03,135 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:03,143 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=325 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/9acc26b83d444fe29f28a852539e50fe 2023-04-08 02:56:03,156 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=325 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/334a0f61eb854ee0b32c0c685479e80b 2023-04-08 02:56:03,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680922563171, max timestamp=1680922563184Num uniques 859; 2023-04-08 02:56:03,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680922563171, max timestamp=1680922563184Num uniques 859; 2023-04-08 02:56:03,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-08 02:56:03,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-08 02:56:03,189 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680922563171, max timestamp=1680922563184Num uniques 859; 2023-04-08 02:56:03,191 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-08 02:56:03,193 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=325 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/c52f27e7527f46ec987375d1fd144d09 2023-04-08 02:56:03,197 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/9acc26b83d444fe29f28a852539e50fe as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/9acc26b83d444fe29f28a852539e50fe 2023-04-08 02:56:03,200 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/9acc26b83d444fe29f28a852539e50fe, entries=250, sequenceid=325, filesize=16.7 K 2023-04-08 02:56:03,201 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/334a0f61eb854ee0b32c0c685479e80b as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/334a0f61eb854ee0b32c0c685479e80b 2023-04-08 02:56:03,204 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:03,205 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:03,206 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/334a0f61eb854ee0b32c0c685479e80b, entries=150, sequenceid=325, filesize=11.9 K 2023-04-08 02:56:03,206 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/c52f27e7527f46ec987375d1fd144d09 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/c52f27e7527f46ec987375d1fd144d09 2023-04-08 02:56:03,208 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:03,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:03,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:03,209 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:03,209 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:03,210 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:03,210 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:03,210 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,210 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,210 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/c52f27e7527f46ec987375d1fd144d09, entries=150, sequenceid=325, filesize=11.9 K 2023-04-08 02:56:03,211 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=60.38 KB/61830 for 61e4c342830c067a5389bf73aa5cd767 in 77ms, sequenceid=325, compaction requested=true 2023-04-08 02:56:03,211 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:03,211 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:03,211 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:03,211 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:56:03,211 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:56:03,211 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:03,212 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:03,212 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,212 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:03,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:03,212 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,212 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:03,212 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,212 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 54272 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:56:03,212 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/A is initiating minor compaction (all files) 2023-04-08 02:56:03,213 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:56:03,213 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/A in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:03,213 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/B is initiating minor compaction (all files) 2023-04-08 02:56:03,213 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,213 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/B in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:03,213 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/934fcc2b5702463a99bb3e32425a2741, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/ac64aff2506a49ff9352fabcbf3cf464, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/1ab86acd67e3408f868aed861ed47827, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/9acc26b83d444fe29f28a852539e50fe] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=53 K 2023-04-08 02:56:03,213 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/712f7aa9f2db45fd9eccdc45367de9f7, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/f5f5aa780fee4142b149509eabbaa3a6, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/fa1e0045333b48a9948c5a7f55e973b4, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/334a0f61eb854ee0b32c0c685479e80b] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=48.2 K 2023-04-08 02:56:03,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:03,213 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,213 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:03,213 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 934fcc2b5702463a99bb3e32425a2741, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1680922560199 2023-04-08 02:56:03,213 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 712f7aa9f2db45fd9eccdc45367de9f7, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1680922560199 2023-04-08 02:56:03,214 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting ac64aff2506a49ff9352fabcbf3cf464, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1680922561344 2023-04-08 02:56:03,214 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting f5f5aa780fee4142b149509eabbaa3a6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1680922561344 2023-04-08 02:56:03,214 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 1ab86acd67e3408f868aed861ed47827, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1680922561998 2023-04-08 02:56:03,214 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting fa1e0045333b48a9948c5a7f55e973b4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1680922561998 2023-04-08 02:56:03,214 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 9acc26b83d444fe29f28a852539e50fe, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1680922563127 2023-04-08 02:56:03,214 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 334a0f61eb854ee0b32c0c685479e80b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1680922563127 2023-04-08 02:56:03,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:56:03,223 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=67.09 KB heapSize=181.08 KB 2023-04-08 02:56:03,223 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:56:03,223 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:03,223 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:56:03,223 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:03,223 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:56:03,223 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:03,225 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#B#compaction#519 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:03,233 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:03,233 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:03,233 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:03,233 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:03,233 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:03,237 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#A#compaction#520 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:03,238 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:03,238 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:03,238 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:03,238 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:03,239 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:03,239 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:03,239 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:03,239 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:03,239 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,240 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:03,240 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:03,242 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,242 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:03,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:03,242 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,242 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:03,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:03,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:03,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:03,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:03,246 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35377bd3[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:03,246 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:03,246 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,248 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,248 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,248 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:03,248 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:03,248 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:03,249 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:03,250 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,250 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:03,250 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:03,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:03,250 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:03,251 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,252 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:03,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,253 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,254 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:03,254 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:03,254 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:03,254 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:03,254 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:03,254 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:03,254 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:03,256 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:03,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:03,257 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,258 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,258 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:03,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,258 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:03,258 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:03,259 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,260 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,261 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,261 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:03,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,262 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:03,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922623255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:03,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922623256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,266 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:03,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 161 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922623258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,267 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,267 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:03,267 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,267 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:03,268 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:03,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:03,268 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:03,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:03,267 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,268 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,268 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,268 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,268 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:03,267 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:03,268 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:03,268 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:03,268 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:03,268 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:03,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922623261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,271 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:03,271 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922623261, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,282 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/c2b5687f151d4091b625909dbec1ab4e as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/c2b5687f151d4091b625909dbec1ab4e 2023-04-08 02:56:03,286 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/41eb067108cf4bb88b28f19d0f50b115 2023-04-08 02:56:03,287 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/2dcc4b54cac54f26b74dcaab2f2ea1e3 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/2dcc4b54cac54f26b74dcaab2f2ea1e3 2023-04-08 02:56:03,289 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/B of 61e4c342830c067a5389bf73aa5cd767 into c2b5687f151d4091b625909dbec1ab4e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:03,290 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:03,290 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/B, priority=12, startTime=1680922563211; duration=0sec 2023-04-08 02:56:03,290 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:03,290 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:56:03,291 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49358 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:56:03,292 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/C is initiating minor compaction (all files) 2023-04-08 02:56:03,292 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/C in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:03,292 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/05fe345c4b3e49bfa7160bd3e07d6e11, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/299b0b60150f41e69e44767b98320346, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/8a1ce1acaec14d8cb7308cb4462fb534, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/c52f27e7527f46ec987375d1fd144d09] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=48.2 K 2023-04-08 02:56:03,293 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 05fe345c4b3e49bfa7160bd3e07d6e11, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=278, earliestPutTs=1680922560199 2023-04-08 02:56:03,293 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/A of 61e4c342830c067a5389bf73aa5cd767 into 2dcc4b54cac54f26b74dcaab2f2ea1e3(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:03,293 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:03,293 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/A, priority=12, startTime=1680922563211; duration=0sec 2023-04-08 02:56:03,293 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 299b0b60150f41e69e44767b98320346, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=294, earliestPutTs=1680922561344 2023-04-08 02:56:03,293 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:03,293 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 8a1ce1acaec14d8cb7308cb4462fb534, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=314, earliestPutTs=1680922561998 2023-04-08 02:56:03,294 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting c52f27e7527f46ec987375d1fd144d09, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1680922563127 2023-04-08 02:56:03,314 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#C#compaction#523 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:03,327 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/0873b50d97de42fd98af374c071d40ab 2023-04-08 02:56:03,368 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:03,369 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:03,370 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:03,371 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:03,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:03,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922623367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,371 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:03,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922623367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,371 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:03,372 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:03,372 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:03,372 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:03,372 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922623367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,372 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:03,372 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:03,373 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,373 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:03,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922623372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:03,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922623372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,374 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:03,374 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:03,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:03,374 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,374 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:03,375 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:03,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,375 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:03,375 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:03,375 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:03,376 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:03,376 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:03,376 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:03,376 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:03,376 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:03,376 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:03,376 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:03,376 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:03,376 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:03,376 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:03,376 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:03,376 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,376 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:03,376 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:03,378 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,378 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:03,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:03,383 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,383 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,383 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:03,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:03,384 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4f086675[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:03,384 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:03,384 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,385 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,385 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:03,385 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:03,385 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:03,386 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,386 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:03,390 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,390 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:03,390 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:03,390 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:03,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:03,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:03,392 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:03,392 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:03,392 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:03,392 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:03,392 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:03,392 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:03,394 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,394 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:03,394 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,394 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:03,395 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:03,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,396 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,397 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:03,398 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,398 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,398 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:03,399 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:03,399 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:03,399 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,398 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:03,399 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:03,399 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:03,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:03,399 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:03,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,399 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,399 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:03,399 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,399 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,399 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:03,399 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:03,400 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:03,499 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:03,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:03,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:03,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:03,503 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:03,503 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:03,503 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:03,503 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:03,503 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:03,504 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,504 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,505 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:03,505 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:03,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:03,506 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,506 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,506 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:03,506 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:03,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,507 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:03,507 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:03,507 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:03,507 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:03,507 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:03,507 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:03,508 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:03,508 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:03,508 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:03,508 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:03,508 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:03,511 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,511 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:03,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:03,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:03,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:03,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:03,513 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,513 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:03,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:03,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1c79de0e[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:03,513 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:03,514 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,515 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,515 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:03,515 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:03,515 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:03,515 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,515 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:03,516 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:03,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:03,517 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:03,517 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:03,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:03,517 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:03,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:03,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:03,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:03,520 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:03,520 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:03,520 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:03,520 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:03,520 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:03,522 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,522 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:03,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,522 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:03,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:03,522 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:03,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:03,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:03,524 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,527 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:03,527 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:03,527 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:03,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:03,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:03,527 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,528 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,528 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,528 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,528 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,528 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:03,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:03,527 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:03,528 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:03,527 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:03,528 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,528 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,528 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:03,528 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:03,528 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:03,529 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:03,576 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:03,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922623573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:03,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922623573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:03,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922623573, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,577 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:03,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 162 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922623574, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,579 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:03,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922623575, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,628 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:03,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:03,631 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:03,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:03,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:03,641 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:03,641 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:03,641 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:03,641 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:03,641 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,641 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,643 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,643 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:03,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:03,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:03,643 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,643 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:03,644 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:03,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,644 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:03,662 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:03,662 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:03,662 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:03,662 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:03,662 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:03,664 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:03,664 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:03,664 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:03,664 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:03,664 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:03,664 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:03,664 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:03,664 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:03,664 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,665 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:03,665 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:03,667 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,667 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:03,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:03,667 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,667 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:03,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:03,667 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:03,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,668 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:03,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:03,669 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4b8d96fe[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:03,669 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:03,669 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,670 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,670 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:03,670 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,670 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:03,670 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:03,670 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:03,673 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:03,673 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:03,673 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:03,674 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:03,674 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:03,675 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:03,677 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:03,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,678 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:03,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,681 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:03,681 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:03,681 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:03,681 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:03,681 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:03,681 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:03,681 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:03,684 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,684 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:03,684 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,685 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:03,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:03,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,690 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:03,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:03,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:03,691 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,691 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:03,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,692 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:03,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,699 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,699 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,699 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,699 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:03,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:03,699 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:03,699 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:03,699 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,699 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:03,699 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:03,699 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:03,699 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,699 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:03,699 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,699 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:03,700 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:03,700 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:03,730 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/b10816d244f24d81971994b53cf28ae7 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/b10816d244f24d81971994b53cf28ae7 2023-04-08 02:56:03,735 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/C of 61e4c342830c067a5389bf73aa5cd767 into b10816d244f24d81971994b53cf28ae7(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:03,735 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:03,735 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/C, priority=12, startTime=1680922563211; duration=0sec 2023-04-08 02:56:03,735 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:03,746 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=352 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/f1e2080c659c4719a448fffe5665cd89 2023-04-08 02:56:03,755 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/41eb067108cf4bb88b28f19d0f50b115 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/41eb067108cf4bb88b28f19d0f50b115 2023-04-08 02:56:03,764 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/41eb067108cf4bb88b28f19d0f50b115, entries=250, sequenceid=352, filesize=16.7 K 2023-04-08 02:56:03,765 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/0873b50d97de42fd98af374c071d40ab as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/0873b50d97de42fd98af374c071d40ab 2023-04-08 02:56:03,777 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/0873b50d97de42fd98af374c071d40ab, entries=150, sequenceid=352, filesize=11.9 K 2023-04-08 02:56:03,778 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/f1e2080c659c4719a448fffe5665cd89 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/f1e2080c659c4719a448fffe5665cd89 2023-04-08 02:56:03,789 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/f1e2080c659c4719a448fffe5665cd89, entries=150, sequenceid=352, filesize=11.9 K 2023-04-08 02:56:03,790 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~200.95 KB/205776, currentSize=107.34 KB/109920 for 61e4c342830c067a5389bf73aa5cd767 in 568ms, sequenceid=352, compaction requested=false 2023-04-08 02:56:03,790 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:03,799 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:03,800 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:03,803 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:03,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:03,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:03,808 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:03,808 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:03,808 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:03,808 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:03,809 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,809 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,812 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:03,812 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:03,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:03,812 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,813 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,813 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:03,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,813 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:03,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,814 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:03,814 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:03,814 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:03,814 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:03,814 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:03,814 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:03,816 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:03,816 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:03,816 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:03,816 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:03,821 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-08 02:56:03,821 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:56:03,821 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:03,822 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:56:03,822 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:03,822 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:56:03,822 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:03,850 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/88606825bf214b25a7100a5b9b10d7d9 2023-04-08 02:56:03,868 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/fe65f867d34b4913abfa942f15654108 2023-04-08 02:56:03,886 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:56:03,886 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:03,900 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=374 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/b6815f1d6c884fb38b1d3a6c08af233c 2023-04-08 02:56:03,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:03,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922623894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,901 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:03,901 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:03,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 82 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922623894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922623894, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,902 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:03,902 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:03,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922623895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,902 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922623896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,904 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/88606825bf214b25a7100a5b9b10d7d9 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/88606825bf214b25a7100a5b9b10d7d9 2023-04-08 02:56:03,912 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/88606825bf214b25a7100a5b9b10d7d9, entries=150, sequenceid=374, filesize=11.9 K 2023-04-08 02:56:03,913 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/fe65f867d34b4913abfa942f15654108 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/fe65f867d34b4913abfa942f15654108 2023-04-08 02:56:03,917 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/fe65f867d34b4913abfa942f15654108, entries=150, sequenceid=374, filesize=11.9 K 2023-04-08 02:56:03,918 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/b6815f1d6c884fb38b1d3a6c08af233c as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/b6815f1d6c884fb38b1d3a6c08af233c 2023-04-08 02:56:03,922 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/b6815f1d6c884fb38b1d3a6c08af233c, entries=150, sequenceid=374, filesize=11.9 K 2023-04-08 02:56:03,923 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 61e4c342830c067a5389bf73aa5cd767 in 102ms, sequenceid=374, compaction requested=true 2023-04-08 02:56:03,923 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:03,923 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:03,923 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:03,923 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:03,923 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,923 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:03,923 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:03,925 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,925 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,926 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,926 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:03,926 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:03,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:03,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:03,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:03,927 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:03,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:03,929 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@45ea85bf[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:03,929 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:03,929 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,930 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,930 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,931 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,931 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:03,931 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:03,931 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:03,931 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,931 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:03,932 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:03,932 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:03,932 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:03,933 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:03,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:03,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:03,935 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:03,936 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:03,936 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:03,936 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:03,936 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:03,938 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,938 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:03,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,939 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:03,939 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:03,939 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:03,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,941 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:03,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:03,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,943 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,943 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,943 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:03,943 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:03,943 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,943 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:03,943 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:03,944 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:03,944 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:03,944 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:03,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:03,944 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:03,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:03,944 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:03,944 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:03,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:03,945 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:03,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:03,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:03,945 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:04,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:56:04,006 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-08 02:56:04,006 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:56:04,006 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:04,006 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:56:04,006 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:04,006 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:56:04,006 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:04,014 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=388 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/88ef453cb6c445028435b771b1807763 2023-04-08 02:56:04,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:04,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922624030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:04,039 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922624030, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,039 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:04,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922624031, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,040 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:04,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50216 deadline: 1680922624032, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,045 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:04,045 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 172 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922624035, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,045 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:04,046 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:04,047 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:04,048 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:04,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:04,049 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:04,049 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:04,049 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:04,049 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:04,049 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,049 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,051 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:04,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:04,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:04,051 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,051 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,051 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:04,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,052 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:04,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,056 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:04,060 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:04,060 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:04,060 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:04,060 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:04,060 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:04,067 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:04,067 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:04,067 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:04,067 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:04,067 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:04,067 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:04,067 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:04,067 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:04,067 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,067 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:04,067 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:04,069 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,069 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,069 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:04,072 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:04,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:04,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:04,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:04,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:04,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:04,074 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@49dd78b9[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:04,074 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:04,074 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,076 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,076 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,076 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:04,076 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:04,076 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:04,076 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,076 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:04,078 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:04,078 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:04,078 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:04,079 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:04,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:04,079 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:04,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:04,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,084 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:04,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:04,086 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:04,086 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:04,086 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:04,086 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:04,086 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:04,086 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:04,088 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,088 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:04,088 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:04,088 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:04,088 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:04,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:04,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:04,089 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,089 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:04,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,090 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:04,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,093 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:04,093 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:04,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:04,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:04,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:04,093 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:04,094 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:04,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:04,093 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:04,094 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,094 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,094 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,094 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:04,094 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:04,094 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,094 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,094 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,094 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:04,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:04,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922624140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:04,148 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922624140, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,148 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:04,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 173 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922624141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:04,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922624141, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,149 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:04,149 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922624146, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,194 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:04,195 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:04,197 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:04,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:04,200 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:04,200 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:04,200 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:04,200 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:04,200 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:04,200 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,200 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,203 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:04,203 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:04,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:04,203 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,203 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:04,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,204 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:04,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,204 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:04,208 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:04,208 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:04,208 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:04,208 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:04,208 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:04,208 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:04,208 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:04,210 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:04,210 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:04,210 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:04,210 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:04,210 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:04,210 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:04,210 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,210 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:04,210 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:04,216 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,216 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:04,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:04,217 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,217 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:04,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:04,218 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:04,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,219 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:04,219 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:04,220 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1b065622[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:04,220 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:04,220 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,221 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,221 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,221 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,221 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:04,221 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:04,221 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:04,221 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:04,224 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:04,224 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:04,224 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:04,224 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:04,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:04,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:04,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:04,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,227 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:04,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:04,229 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:04,229 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:04,229 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:04,229 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:04,229 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:04,229 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:04,232 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,232 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:04,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,232 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:04,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:04,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,239 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:04,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:04,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:04,241 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:04,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,243 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:04,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,246 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:04,246 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:04,246 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:04,246 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:04,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:04,246 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:04,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:04,246 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,246 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,246 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,246 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,246 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:04,246 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:04,246 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,246 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,247 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:04,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:04,347 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:04,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:04,349 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:04,350 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:04,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:04,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:04,350 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 175 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922624350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,350 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:04,351 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:04,351 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:04,351 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:04,351 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,351 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,353 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,353 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:04,353 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:04,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:04,353 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,353 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:04,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,353 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:04,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922624350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:04,354 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922624350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,354 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:04,354 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922624350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:04,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,354 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:04,354 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:04,354 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:04,354 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:04,354 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:04,355 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:04,355 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:04,355 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:04,355 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:04,355 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922624350, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,355 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:04,355 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:04,355 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:04,355 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:04,355 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:04,355 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:04,355 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,355 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:04,355 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:04,364 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:04,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:04,364 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,364 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,364 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:04,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:04,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:04,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:04,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:04,366 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3c9b09da[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:04,366 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:04,366 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,367 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,368 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:04,368 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:04,368 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,368 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:04,368 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:04,371 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,371 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:04,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:04,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:04,371 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:04,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:04,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:04,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:04,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:04,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:04,375 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:04,375 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:04,375 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:04,375 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:04,375 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:04,375 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:04,377 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:04,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:04,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:04,378 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,378 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,378 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:04,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,378 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:04,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:04,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:04,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,380 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:04,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,383 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,383 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:04,383 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:04,383 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:04,383 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,383 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:04,384 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:04,383 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:04,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:04,383 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:04,384 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,384 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:04,384 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,384 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,384 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,384 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,384 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:04,384 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:04,384 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:04,431 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=388 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/a753fc395b264beab8430cf5e2da7c32 2023-04-08 02:56:04,484 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:04,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:04,486 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:04,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:04,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:04,491 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:04,492 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:04,492 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:04,492 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:04,492 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,492 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,494 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,494 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:04,494 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:04,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:04,494 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,494 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:04,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,494 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:04,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,495 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:04,496 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:04,496 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:04,496 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:04,496 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:04,496 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:04,498 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:04,498 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:04,498 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:04,498 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:04,498 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:04,498 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:04,498 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:04,498 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:04,498 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,498 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:04,498 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:04,500 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:04,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:04,500 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:04,501 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,501 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:04,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:04,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:04,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:04,505 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2d4da48b[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:04,505 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:04,505 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,506 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,507 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,507 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:04,507 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:04,507 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:04,507 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,507 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:04,509 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:04,509 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:04,509 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:04,509 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:04,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:04,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:04,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:04,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:04,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:04,513 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:04,513 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:04,513 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:04,514 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:04,514 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:04,514 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:04,515 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,515 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:04,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:04,515 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,515 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:04,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:04,516 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:04,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:04,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:04,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,520 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:04,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,522 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,522 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,522 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,522 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,522 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:04,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:04,522 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:04,523 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:04,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:04,523 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:04,522 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,522 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:04,523 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,523 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:04,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:04,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:04,523 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:04,523 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:04,623 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:04,624 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:04,625 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:04,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:04,626 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:04,626 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:04,627 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:04,627 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:04,627 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:04,628 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,628 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,629 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:04,629 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:04,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:04,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,630 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:04,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,631 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:04,631 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:04,631 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:04,631 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:04,631 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:04,631 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:04,631 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:04,631 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:04,631 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:04,631 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:04,632 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:04,632 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:04,632 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:04,632 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:04,632 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,632 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:04,632 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:04,632 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,633 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,633 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,634 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:04,636 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,636 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:04,636 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@564c1dab[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:04,636 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:04,636 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:04,636 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,638 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,638 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,638 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,638 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:04,638 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:04,638 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:04,638 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:04,641 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:04,641 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:04,641 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,641 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:04,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:04,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:04,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:04,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:04,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:04,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:04,643 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:04,643 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:04,643 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:04,643 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:04,643 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:04,643 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:04,645 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,645 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:04,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,645 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:04,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:04,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:04,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:04,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:04,646 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:04,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,647 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:04,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,649 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:04,649 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:04,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:04,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:04,649 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:04,649 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:04,649 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,649 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,649 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,649 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,649 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,649 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,649 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,649 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:04,649 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:04,649 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:04,649 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:04,649 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:04,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:04,654 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:04,654 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 177 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922624651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:04,659 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922624654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,659 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:04,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922624655, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,660 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:04,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922624657, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,662 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:04,662 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922624658, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,750 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:04,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:04,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:04,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:04,756 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:04,756 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:04,756 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:04,756 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:04,756 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:04,756 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,757 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,758 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:04,758 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:04,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:04,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,759 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:04,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:04,759 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,759 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:04,759 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:04,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:04,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:04,759 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,760 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:04,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:04,760 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:04,760 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:04,760 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:04,760 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:04,760 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:04,760 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:04,760 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:04,760 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:04,760 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,760 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:04,760 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:04,763 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,763 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:04,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:04,763 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,763 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:04,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:04,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:04,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:04,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:04,766 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@ff7192b[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:04,766 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:04,766 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,771 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,771 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:04,771 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:04,771 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,772 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:04,772 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:04,773 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,773 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:04,773 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:04,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:04,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:04,773 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:04,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:04,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:04,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:04,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:04,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:04,777 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:04,777 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:04,777 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:04,777 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:04,777 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:04,781 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,781 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:04,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:04,781 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:04,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,783 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:04,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,785 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:04,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:04,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,788 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,788 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,788 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,788 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,788 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,788 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:04,788 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,788 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:04,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:04,788 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:04,789 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:04,789 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:04,789 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:04,789 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:04,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:04,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:04,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:04,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:04,844 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=388 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/bf3322389c0e4b39abdff279f49a6bc9 2023-04-08 02:56:04,848 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/88ef453cb6c445028435b771b1807763 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/88ef453cb6c445028435b771b1807763 2023-04-08 02:56:04,853 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/88ef453cb6c445028435b771b1807763, entries=200, sequenceid=388, filesize=14.3 K 2023-04-08 02:56:04,854 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/a753fc395b264beab8430cf5e2da7c32 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/a753fc395b264beab8430cf5e2da7c32 2023-04-08 02:56:04,860 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/a753fc395b264beab8430cf5e2da7c32, entries=150, sequenceid=388, filesize=11.9 K 2023-04-08 02:56:04,861 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/bf3322389c0e4b39abdff279f49a6bc9 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/bf3322389c0e4b39abdff279f49a6bc9 2023-04-08 02:56:04,865 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/bf3322389c0e4b39abdff279f49a6bc9, entries=150, sequenceid=388, filesize=11.9 K 2023-04-08 02:56:04,866 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~219.84 KB/225120, currentSize=100.63 KB/103050 for 61e4c342830c067a5389bf73aa5cd767 in 860ms, sequenceid=388, compaction requested=true 2023-04-08 02:56:04,866 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:04,866 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:04,866 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:04,866 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:56:04,866 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:56:04,866 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:04,868 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:56:04,868 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/B is initiating minor compaction (all files) 2023-04-08 02:56:04,868 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/B in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:04,868 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/c2b5687f151d4091b625909dbec1ab4e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/0873b50d97de42fd98af374c071d40ab, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/fe65f867d34b4913abfa942f15654108, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/a753fc395b264beab8430cf5e2da7c32] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=48.3 K 2023-04-08 02:56:04,869 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 56848 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:56:04,869 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/A is initiating minor compaction (all files) 2023-04-08 02:56:04,869 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/A in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:04,869 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/2dcc4b54cac54f26b74dcaab2f2ea1e3, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/41eb067108cf4bb88b28f19d0f50b115, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/88606825bf214b25a7100a5b9b10d7d9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/88ef453cb6c445028435b771b1807763] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=55.5 K 2023-04-08 02:56:04,869 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting c2b5687f151d4091b625909dbec1ab4e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1680922563127 2023-04-08 02:56:04,869 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 2dcc4b54cac54f26b74dcaab2f2ea1e3, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1680922563127 2023-04-08 02:56:04,870 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 0873b50d97de42fd98af374c071d40ab, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1680922563184 2023-04-08 02:56:04,870 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 41eb067108cf4bb88b28f19d0f50b115, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1680922563184 2023-04-08 02:56:04,870 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting fe65f867d34b4913abfa942f15654108, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1680922563253 2023-04-08 02:56:04,870 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 88606825bf214b25a7100a5b9b10d7d9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1680922563253 2023-04-08 02:56:04,872 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 88ef453cb6c445028435b771b1807763, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1680922563892 2023-04-08 02:56:04,872 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting a753fc395b264beab8430cf5e2da7c32, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1680922563892 2023-04-08 02:56:04,883 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#A#compaction#531 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:04,885 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#B#compaction#532 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:04,889 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:04,889 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:04,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:04,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:04,892 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:04,895 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:04,895 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:04,895 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:04,895 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:04,896 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,896 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,897 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,897 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:04,897 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:04,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:04,897 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:04,897 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:04,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,898 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:04,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:04,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:04,898 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:04,900 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:04,900 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:04,900 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:04,900 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:04,900 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:04,901 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:04,901 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:04,902 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:04,902 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:04,902 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-08 02:56:04,902 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:56:04,902 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:04,902 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:56:04,903 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:04,903 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:56:04,903 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:04,910 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/446d87822356458b8afaba0ad7499add as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/446d87822356458b8afaba0ad7499add 2023-04-08 02:56:04,918 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/A of 61e4c342830c067a5389bf73aa5cd767 into 446d87822356458b8afaba0ad7499add(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:04,919 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:04,919 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/A, priority=12, startTime=1680922564866; duration=0sec 2023-04-08 02:56:04,919 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:04,919 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:56:04,920 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 49494 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:56:04,920 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/C is initiating minor compaction (all files) 2023-04-08 02:56:04,920 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/C in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:04,920 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/b10816d244f24d81971994b53cf28ae7, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/f1e2080c659c4719a448fffe5665cd89, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/b6815f1d6c884fb38b1d3a6c08af233c, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/bf3322389c0e4b39abdff279f49a6bc9] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=48.3 K 2023-04-08 02:56:04,921 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting b10816d244f24d81971994b53cf28ae7, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=325, earliestPutTs=1680922563127 2023-04-08 02:56:04,921 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting f1e2080c659c4719a448fffe5665cd89, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=352, earliestPutTs=1680922563184 2023-04-08 02:56:04,922 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting b6815f1d6c884fb38b1d3a6c08af233c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=374, earliestPutTs=1680922563253 2023-04-08 02:56:04,922 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting bf3322389c0e4b39abdff279f49a6bc9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1680922563892 2023-04-08 02:56:04,931 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#C#compaction#534 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:05,163 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:56:05,163 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:05,180 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:05,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922625171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:05,180 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:05,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 136 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922625174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:05,183 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:05,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922625174, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:05,183 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:05,183 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 97 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922625175, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:05,188 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:05,188 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922625180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:05,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:05,287 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922625281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:05,287 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:05,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922625281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:05,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:05,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 99 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922625284, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:05,288 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:05,288 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922625286, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:05,298 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:05,298 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922625289, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:05,327 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/fb6f3f9f39914f56b2a0323bc6568d59 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/fb6f3f9f39914f56b2a0323bc6568d59 2023-04-08 02:56:05,332 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/B of 61e4c342830c067a5389bf73aa5cd767 into fb6f3f9f39914f56b2a0323bc6568d59(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:05,332 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:05,332 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/B, priority=12, startTime=1680922564866; duration=0sec 2023-04-08 02:56:05,332 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:05,336 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/d97e3390bb054efd910a418a48d11cf5 2023-04-08 02:56:05,350 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/e1116fb2355b449183d931c3806042af as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/e1116fb2355b449183d931c3806042af 2023-04-08 02:56:05,354 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/C of 61e4c342830c067a5389bf73aa5cd767 into e1116fb2355b449183d931c3806042af(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:05,354 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:05,354 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/C, priority=12, startTime=1680922564866; duration=0sec 2023-04-08 02:56:05,354 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:05,491 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:05,492 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922625489, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:05,493 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:05,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922625490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:05,493 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:05,493 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922625490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:05,493 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:05,494 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922625490, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:05,502 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:05,502 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922625500, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:05,748 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/ee5fb1e8201440c7a18bfc4761d332fd 2023-04-08 02:56:05,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:05,798 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922625796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:05,798 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:05,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922625796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:05,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:05,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922625796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:05,799 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:05,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922625796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:05,807 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:05,807 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922625805, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,156 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=406 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/effdd6c6427d46629022ab42084b45e5 2023-04-08 02:56:06,160 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/d97e3390bb054efd910a418a48d11cf5 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/d97e3390bb054efd910a418a48d11cf5 2023-04-08 02:56:06,164 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/d97e3390bb054efd910a418a48d11cf5, entries=150, sequenceid=406, filesize=11.9 K 2023-04-08 02:56:06,172 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/ee5fb1e8201440c7a18bfc4761d332fd as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/ee5fb1e8201440c7a18bfc4761d332fd 2023-04-08 02:56:06,177 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/ee5fb1e8201440c7a18bfc4761d332fd, entries=150, sequenceid=406, filesize=11.9 K 2023-04-08 02:56:06,177 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/effdd6c6427d46629022ab42084b45e5 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/effdd6c6427d46629022ab42084b45e5 2023-04-08 02:56:06,180 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/effdd6c6427d46629022ab42084b45e5, entries=150, sequenceid=406, filesize=11.9 K 2023-04-08 02:56:06,181 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=73.80 KB/75570 for 61e4c342830c067a5389bf73aa5cd767 in 1279ms, sequenceid=406, compaction requested=false 2023-04-08 02:56:06,181 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:06,181 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:06,181 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:06,181 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:06,181 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,184 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:06,184 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:06,186 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,186 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:06,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:06,186 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,186 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:06,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:06,186 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:06,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:06,187 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:06,188 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2669e9b1[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:06,188 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:06,188 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,191 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,191 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,192 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:06,191 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:06,192 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:06,192 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:06,193 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:06,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:06,194 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:06,194 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:06,194 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:06,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:06,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:06,198 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:06,198 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:06,199 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:06,199 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:06,199 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:06,199 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:06,200 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,200 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:06,200 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,200 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:06,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:06,201 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:06,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:06,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:06,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,201 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:06,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,203 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:06,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,205 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:06,205 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:06,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:06,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:06,206 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:06,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:06,206 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,206 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:06,206 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:06,206 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,206 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,206 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,206 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,206 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:06,206 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:06,206 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:06,207 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:06,207 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:06,207 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:06,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:56:06,304 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-08 02:56:06,304 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:56:06,304 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:06,304 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:56:06,304 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:06,304 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:56:06,305 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:06,307 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:06,308 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:06,311 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:06,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:06,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:06,313 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:06,313 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:06,313 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:06,313 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:06,314 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,314 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:06,316 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,316 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:06,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:06,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:06,317 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,317 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:06,317 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:06,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:06,317 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:06,319 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:06,319 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:06,319 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:06,319 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:06,319 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:06,319 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:06,319 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:06,319 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:06,319 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:06,319 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:06,319 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:06,320 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:06,320 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:06,320 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,320 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:06,320 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:06,321 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:06,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:06,321 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,321 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/694bfa1098de42319dd25c89dbf90949 2023-04-08 02:56:06,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:06,322 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,322 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:06,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:06,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:06,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:06,324 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@42d8e17f[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:06,324 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:06,324 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,325 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,325 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,325 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,325 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:06,325 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:06,325 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:06,326 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,326 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:06,327 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:06,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:06,328 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:06,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:06,328 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:06,328 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:06,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:06,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:06,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:06,330 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:06,330 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:06,330 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:06,331 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:06,331 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:06,331 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:06,333 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,333 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:06,333 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,334 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:06,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:06,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:06,334 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:06,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:06,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:06,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 108 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922626325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:06,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:06,335 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922626325, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,335 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:06,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922626327, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:06,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:06,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922626328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,338 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:06,339 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922626330, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,341 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/fa7d34b48ffb4bcf9326d25c4fb7e81a 2023-04-08 02:56:06,342 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,342 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:06,342 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:06,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:06,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:06,342 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:06,342 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:06,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:06,342 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:06,342 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:06,342 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:06,342 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:06,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:06,342 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,342 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:06,342 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,342 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,342 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,437 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:06,437 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922626436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:06,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 150 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922626436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:06,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 110 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922626436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,442 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:06,442 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:06,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922626439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,443 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:06,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922626439, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,443 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:06,444 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:06,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:06,445 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:06,445 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:06,446 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:06,446 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:06,446 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:06,446 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,446 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:06,447 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:06,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:06,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:06,447 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:06,448 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,448 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:06,448 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:06,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:06,448 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:06,449 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:06,449 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:06,449 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:06,449 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:06,449 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:06,450 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:06,450 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:06,450 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:06,450 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:06,450 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:06,450 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:06,450 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:06,450 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:06,450 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,450 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:06,450 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:06,452 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,452 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:06,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:06,452 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,452 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:06,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:06,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:06,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:06,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:06,455 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2717c53[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:06,455 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:06,455 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,457 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,457 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,457 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,457 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:06,457 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:06,457 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:06,458 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,458 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:06,462 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,462 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:06,462 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:06,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:06,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:06,462 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:06,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:06,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:06,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:06,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:06,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:06,464 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:06,464 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:06,464 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:06,465 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:06,465 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:06,466 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,466 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,466 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:06,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:06,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:06,467 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,467 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:06,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:06,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:06,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:06,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,469 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:06,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,472 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,472 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:06,472 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:06,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:06,472 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:06,472 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:06,472 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:06,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:06,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:06,472 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:06,473 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,473 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,473 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,473 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,472 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:06,473 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:06,473 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:06,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:06,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:06,573 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:06,573 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:06,575 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:06,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:06,576 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:06,576 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:06,576 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:06,576 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:06,576 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:06,576 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,576 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:06,578 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,578 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:06,578 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:06,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:06,578 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,578 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:06,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:06,578 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:06,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:06,579 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:06,579 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:06,579 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:06,579 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:06,579 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:06,579 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:06,579 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:06,579 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:06,579 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:06,579 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:06,580 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:06,580 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:06,580 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:06,580 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:06,580 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,580 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:06,580 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:06,581 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:06,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:06,582 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:06,582 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,582 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:06,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:06,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:06,583 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:06,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@35f81377[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:06,584 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:06,584 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,585 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,585 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:06,585 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:06,585 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:06,585 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,585 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:06,587 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,587 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:06,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:06,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:06,587 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:06,587 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:06,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:06,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:06,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:06,588 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:06,589 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:06,589 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:06,589 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:06,589 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:06,589 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:06,589 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:06,591 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,591 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:06,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,591 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,591 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:06,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:06,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:06,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,591 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:06,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:06,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:06,593 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:06,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,595 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,595 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:06,595 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:06,595 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,595 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,595 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,595 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:06,595 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:06,596 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:06,596 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:06,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:06,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:06,596 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:06,596 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:06,595 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:06,596 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:06,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:06,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:06,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:06,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922626638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,643 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:06,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922626639, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,644 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:06,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 112 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922626640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:06,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922626645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,651 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:06,651 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922626646, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,696 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:06,697 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:06,699 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:06,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:06,701 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:06,701 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:06,701 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:06,701 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:06,701 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:06,701 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,701 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:06,703 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:06,703 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,703 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:06,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:06,703 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,703 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:06,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:06,704 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:06,704 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:06,704 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:06,704 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:06,704 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:06,704 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:06,705 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:06,705 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:06,705 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:06,705 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:06,705 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:06,705 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:06,706 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:06,706 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:06,706 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:06,706 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:06,706 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,706 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:06,706 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:06,709 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,709 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:06,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:06,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:06,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:06,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:06,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:06,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:06,716 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5cb55018[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:06,716 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:06,716 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,718 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,718 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,718 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:06,718 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:06,718 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:06,718 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,718 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:06,721 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:06,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:06,721 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:06,721 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:06,721 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:06,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:06,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:06,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:06,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:06,724 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:06,724 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:06,724 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:06,724 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:06,724 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:06,724 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:06,738 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,738 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:06,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:06,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:06,739 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,739 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:06,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,740 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,740 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,742 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:06,744 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,744 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:06,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,744 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:06,744 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:06,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,745 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,746 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,748 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:06,748 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:06,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:06,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:06,748 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,748 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:06,748 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:06,748 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,748 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,748 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,748 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:06,748 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:06,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:06,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:06,749 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:06,749 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:06,748 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,748 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:06,750 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:06,755 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=424 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/032f25bdab4c4519b0e460234e736fd0 2023-04-08 02:56:06,761 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/694bfa1098de42319dd25c89dbf90949 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/694bfa1098de42319dd25c89dbf90949 2023-04-08 02:56:06,767 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/694bfa1098de42319dd25c89dbf90949, entries=200, sequenceid=424, filesize=14.3 K 2023-04-08 02:56:06,768 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/fa7d34b48ffb4bcf9326d25c4fb7e81a as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/fa7d34b48ffb4bcf9326d25c4fb7e81a 2023-04-08 02:56:06,772 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/fa7d34b48ffb4bcf9326d25c4fb7e81a, entries=150, sequenceid=424, filesize=11.9 K 2023-04-08 02:56:06,772 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/032f25bdab4c4519b0e460234e736fd0 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/032f25bdab4c4519b0e460234e736fd0 2023-04-08 02:56:06,777 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/032f25bdab4c4519b0e460234e736fd0, entries=150, sequenceid=424, filesize=11.9 K 2023-04-08 02:56:06,778 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 61e4c342830c067a5389bf73aa5cd767 in 474ms, sequenceid=424, compaction requested=true 2023-04-08 02:56:06,778 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:06,778 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:06,778 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:06,778 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:56:06,778 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:06,778 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:56:06,779 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 39918 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:56:06,779 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/A is initiating minor compaction (all files) 2023-04-08 02:56:06,779 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:56:06,779 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/A in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:06,779 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/B is initiating minor compaction (all files) 2023-04-08 02:56:06,779 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/446d87822356458b8afaba0ad7499add, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/d97e3390bb054efd910a418a48d11cf5, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/694bfa1098de42319dd25c89dbf90949] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=39.0 K 2023-04-08 02:56:06,779 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/B in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:06,779 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/fb6f3f9f39914f56b2a0323bc6568d59, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/ee5fb1e8201440c7a18bfc4761d332fd, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/fa7d34b48ffb4bcf9326d25c4fb7e81a] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=36.6 K 2023-04-08 02:56:06,780 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 446d87822356458b8afaba0ad7499add, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1680922563892 2023-04-08 02:56:06,780 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting fb6f3f9f39914f56b2a0323bc6568d59, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1680922563892 2023-04-08 02:56:06,780 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting d97e3390bb054efd910a418a48d11cf5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680922564018 2023-04-08 02:56:06,780 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting ee5fb1e8201440c7a18bfc4761d332fd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680922564018 2023-04-08 02:56:06,780 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 694bfa1098de42319dd25c89dbf90949, keycount=200, bloomtype=ROW, size=14.3 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1680922565172 2023-04-08 02:56:06,781 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting fa7d34b48ffb4bcf9326d25c4fb7e81a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1680922565172 2023-04-08 02:56:06,786 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#A#compaction#540 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:06,789 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#B#compaction#541 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:06,810 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/63349a52891d4ae0afac176b2320d1c2 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/63349a52891d4ae0afac176b2320d1c2 2023-04-08 02:56:06,821 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/B of 61e4c342830c067a5389bf73aa5cd767 into 63349a52891d4ae0afac176b2320d1c2(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:06,821 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:06,821 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/B, priority=13, startTime=1680922566778; duration=0sec 2023-04-08 02:56:06,821 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:06,821 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:56:06,821 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:56:06,822 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/C is initiating minor compaction (all files) 2023-04-08 02:56:06,822 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/C in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:06,822 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/e1116fb2355b449183d931c3806042af, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/effdd6c6427d46629022ab42084b45e5, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/032f25bdab4c4519b0e460234e736fd0] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=36.6 K 2023-04-08 02:56:06,822 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting e1116fb2355b449183d931c3806042af, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=388, earliestPutTs=1680922563892 2023-04-08 02:56:06,822 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting effdd6c6427d46629022ab42084b45e5, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=406, earliestPutTs=1680922564018 2023-04-08 02:56:06,823 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 032f25bdab4c4519b0e460234e736fd0, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1680922565172 2023-04-08 02:56:06,827 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#C#compaction#542 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:06,840 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/5e6ea109808e48b3a778a6587d5dce3b as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/5e6ea109808e48b3a778a6587d5dce3b 2023-04-08 02:56:06,843 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/C of 61e4c342830c067a5389bf73aa5cd767 into 5e6ea109808e48b3a778a6587d5dce3b(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:06,843 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:06,843 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/C, priority=13, startTime=1680922566778; duration=0sec 2023-04-08 02:56:06,843 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:06,849 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:06,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:06,851 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:06,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:06,852 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:06,852 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:06,852 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:06,852 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:06,852 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:06,852 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,852 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:06,854 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,854 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:06,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:06,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:06,854 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,854 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:06,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:06,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:06,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:06,855 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:06,855 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:06,855 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:06,855 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:06,855 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:06,855 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:06,855 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:06,855 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:06,855 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:06,855 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:06,855 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-08 02:56:06,855 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:56:06,855 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:06,855 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:56:06,856 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:06,856 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:56:06,856 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:06,866 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/c77ecc95ca844a289bbb16fe7ba39096 2023-04-08 02:56:06,875 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/13b2d525ed8340e492d600e4d24be16e 2023-04-08 02:56:06,882 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=443 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/5c9b29e1d2174378b5f43c4988eca0dd 2023-04-08 02:56:06,885 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/c77ecc95ca844a289bbb16fe7ba39096 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/c77ecc95ca844a289bbb16fe7ba39096 2023-04-08 02:56:06,888 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/c77ecc95ca844a289bbb16fe7ba39096, entries=150, sequenceid=443, filesize=11.9 K 2023-04-08 02:56:06,889 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/13b2d525ed8340e492d600e4d24be16e as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/13b2d525ed8340e492d600e4d24be16e 2023-04-08 02:56:06,893 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/13b2d525ed8340e492d600e4d24be16e, entries=150, sequenceid=443, filesize=11.9 K 2023-04-08 02:56:06,893 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/5c9b29e1d2174378b5f43c4988eca0dd as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/5c9b29e1d2174378b5f43c4988eca0dd 2023-04-08 02:56:06,896 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/5c9b29e1d2174378b5f43c4988eca0dd, entries=150, sequenceid=443, filesize=11.9 K 2023-04-08 02:56:06,897 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=0 B/0 for 61e4c342830c067a5389bf73aa5cd767 in 42ms, sequenceid=443, compaction requested=false 2023-04-08 02:56:06,897 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:06,897 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:06,897 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:06,897 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:06,897 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,897 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:06,897 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:06,899 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,899 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:06,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:06,900 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,900 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:06,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:06,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:06,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:06,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:06,903 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@59e391e8[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:06,903 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:06,904 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,905 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,905 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,905 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:06,905 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:06,905 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:06,905 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,905 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:06,907 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,908 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:06,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:06,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:06,908 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:06,908 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:06,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:06,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:06,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:06,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:06,911 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:06,911 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:06,911 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:06,911 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:06,911 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:06,911 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:06,913 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:06,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:06,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:06,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:06,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,915 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,916 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,916 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:06,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:06,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:06,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:06,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:06,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,917 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,921 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:06,921 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,921 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:06,921 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:06,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:06,921 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:06,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:06,921 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:06,921 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:06,922 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:06,922 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,922 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:06,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:06,922 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:06,922 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:06,922 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:06,922 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:06,954 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:56:06,954 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-08 02:56:06,956 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:56:06,956 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:06,956 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:56:06,956 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:06,956 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:56:06,956 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:06,963 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=455 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/004a4112001144159a6c0f713ba20c0b 2023-04-08 02:56:06,979 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=455 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/ca9ed82efeeb42ed9de216ba6eb78a57 2023-04-08 02:56:06,986 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:06,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 159 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922626983, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,992 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:06,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922626984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,992 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:06,992 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922626985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,993 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:06,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922626986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,994 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:06,994 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922626987, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:06,998 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=455 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/38bbd683d46e42b7adf51c3cf05c8514 2023-04-08 02:56:07,003 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/004a4112001144159a6c0f713ba20c0b as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/004a4112001144159a6c0f713ba20c0b 2023-04-08 02:56:07,008 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/004a4112001144159a6c0f713ba20c0b, entries=300, sequenceid=455, filesize=19.0 K 2023-04-08 02:56:07,010 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/ca9ed82efeeb42ed9de216ba6eb78a57 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/ca9ed82efeeb42ed9de216ba6eb78a57 2023-04-08 02:56:07,016 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/ca9ed82efeeb42ed9de216ba6eb78a57, entries=150, sequenceid=455, filesize=11.9 K 2023-04-08 02:56:07,017 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/38bbd683d46e42b7adf51c3cf05c8514 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/38bbd683d46e42b7adf51c3cf05c8514 2023-04-08 02:56:07,022 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:07,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:07,023 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/38bbd683d46e42b7adf51c3cf05c8514, entries=150, sequenceid=455, filesize=11.9 K 2023-04-08 02:56:07,024 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 61e4c342830c067a5389bf73aa5cd767 in 70ms, sequenceid=455, compaction requested=true 2023-04-08 02:56:07,024 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:07,024 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:07,024 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:07,024 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 3 compacting, 2 eligible, 16 blocking 2023-04-08 02:56:07,024 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-08 02:56:07,024 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:07,025 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-08 02:56:07,025 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-08 02:56:07,025 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. because compaction request was cancelled 2023-04-08 02:56:07,025 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:56:07,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:07,025 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:07,025 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:07,026 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:07,026 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:07,026 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:07,026 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:07,026 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:07,026 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:56:07,026 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/B is initiating minor compaction (all files) 2023-04-08 02:56:07,026 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/B in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:07,026 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/63349a52891d4ae0afac176b2320d1c2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/13b2d525ed8340e492d600e4d24be16e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/ca9ed82efeeb42ed9de216ba6eb78a57] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=36.7 K 2023-04-08 02:56:07,027 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 63349a52891d4ae0afac176b2320d1c2, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1680922565172 2023-04-08 02:56:07,027 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 13b2d525ed8340e492d600e4d24be16e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1680922566318 2023-04-08 02:56:07,027 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting ca9ed82efeeb42ed9de216ba6eb78a57, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1680922566952 2023-04-08 02:56:07,028 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:07,028 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,028 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:07,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:07,029 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,029 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:07,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:07,029 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:07,029 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:07,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:07,030 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:07,032 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:07,032 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:07,032 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:07,032 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:07,032 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:07,036 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:07,036 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:07,036 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:07,036 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:07,036 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-08 02:56:07,036 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:56:07,036 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:07,036 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:56:07,036 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:07,037 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:56:07,037 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:07,047 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#B#compaction#549 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:07,066 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/87fc3014923c46a495f0f1ce06c7903b 2023-04-08 02:56:07,086 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/07d9ddd005a34647b693f03943c13aa1 2023-04-08 02:56:07,093 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:56:07,093 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:07,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:07,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922627102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:07,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 124 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922627103, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,112 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:07,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 164 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922627105, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,118 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:07,118 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:07,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922627109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,119 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922627111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,202 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/d3aa99f0dd904c3aa2f361276e674dbf as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/d3aa99f0dd904c3aa2f361276e674dbf 2023-04-08 02:56:07,209 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/A of 61e4c342830c067a5389bf73aa5cd767 into d3aa99f0dd904c3aa2f361276e674dbf(size=12.9 K), total size for store is 43.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:07,209 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:07,209 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/A, priority=13, startTime=1680922566778; duration=0sec 2023-04-08 02:56:07,209 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:07,209 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:56:07,210 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37546 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:56:07,210 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/C is initiating minor compaction (all files) 2023-04-08 02:56:07,210 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/C in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:07,210 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/5e6ea109808e48b3a778a6587d5dce3b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/5c9b29e1d2174378b5f43c4988eca0dd, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/38bbd683d46e42b7adf51c3cf05c8514] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=36.7 K 2023-04-08 02:56:07,211 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 5e6ea109808e48b3a778a6587d5dce3b, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1680922565172 2023-04-08 02:56:07,211 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 5c9b29e1d2174378b5f43c4988eca0dd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1680922566318 2023-04-08 02:56:07,212 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 38bbd683d46e42b7adf51c3cf05c8514, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1680922566952 2023-04-08 02:56:07,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:07,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922627212, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,215 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:07,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 126 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:50238 deadline: 1680922627213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,217 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#C#compaction#553 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:07,219 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:07,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 166 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922627213, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,229 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:07,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 210 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922627227, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,231 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/8967d100ce7445049b6d1303e93567c7 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/8967d100ce7445049b6d1303e93567c7 2023-04-08 02:56:07,233 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:07,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922627229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,237 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/C of 61e4c342830c067a5389bf73aa5cd767 into 8967d100ce7445049b6d1303e93567c7(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:07,237 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:07,237 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/C, priority=13, startTime=1680922567024; duration=0sec 2023-04-08 02:56:07,237 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:07,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:07,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922627416, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,420 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:07,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50238 deadline: 1680922627417, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,425 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:07,425 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 168 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922627421, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,434 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:07,434 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922627431, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,438 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:07,439 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922627435, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,462 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/6f8af884c4484b1b83516f5f0cab1b86 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/6f8af884c4484b1b83516f5f0cab1b86 2023-04-08 02:56:07,468 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/B of 61e4c342830c067a5389bf73aa5cd767 into 6f8af884c4484b1b83516f5f0cab1b86(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:07,468 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:07,468 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/B, priority=13, startTime=1680922567024; duration=0sec 2023-04-08 02:56:07,468 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:07,499 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=475 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/f41318b61a154d7c86ac5621c95f5b54 2023-04-08 02:56:07,504 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/87fc3014923c46a495f0f1ce06c7903b as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/87fc3014923c46a495f0f1ce06c7903b 2023-04-08 02:56:07,508 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/87fc3014923c46a495f0f1ce06c7903b, entries=150, sequenceid=475, filesize=11.9 K 2023-04-08 02:56:07,508 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/07d9ddd005a34647b693f03943c13aa1 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/07d9ddd005a34647b693f03943c13aa1 2023-04-08 02:56:07,512 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/07d9ddd005a34647b693f03943c13aa1, entries=150, sequenceid=475, filesize=11.9 K 2023-04-08 02:56:07,512 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/f41318b61a154d7c86ac5621c95f5b54 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/f41318b61a154d7c86ac5621c95f5b54 2023-04-08 02:56:07,516 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/f41318b61a154d7c86ac5621c95f5b54, entries=150, sequenceid=475, filesize=11.9 K 2023-04-08 02:56:07,516 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=60.38 KB/61830 for 61e4c342830c067a5389bf73aa5cd767 in 480ms, sequenceid=475, compaction requested=true 2023-04-08 02:56:07,517 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:07,517 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:07,517 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:07,517 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:07,517 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,517 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:07,517 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:07,518 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,518 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:07,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:07,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:07,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:07,519 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:07,519 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:07,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:07,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:07,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:07,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:07,520 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5079fd63[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:07,520 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:07,520 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:07,521 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:07,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:07,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:07,522 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:07,522 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:07,522 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:07,522 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,522 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:07,524 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:07,524 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:07,524 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,524 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:07,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:07,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:07,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:07,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:07,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:07,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:07,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:07,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:07,527 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:07,527 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:07,527 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:07,528 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:07,528 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:07,528 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:07,530 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:07,530 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:07,530 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:07,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:07,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:07,530 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:07,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:07,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:07,532 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:07,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:07,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:07,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:07,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:07,533 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:07,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:07,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,535 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:07,535 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:07,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,538 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:07,538 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:07,538 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:07,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:07,539 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,539 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:07,539 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:07,539 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,539 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:07,539 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:07,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:07,539 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:07,539 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:07,540 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:07,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:07,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:07,540 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:07,540 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:07,640 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:07,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:07,642 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:07,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:07,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:07,643 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:07,643 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:07,644 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:07,644 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:07,644 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:07,644 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:07,645 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:07,645 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,645 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:07,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:07,646 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,646 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:07,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:07,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:07,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:07,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:07,647 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:07,647 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:07,647 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:07,647 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:07,647 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:07,647 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:07,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:07,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:07,648 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:07,648 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:07,648 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-08 02:56:07,648 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:56:07,648 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:07,648 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:56:07,649 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:07,649 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:56:07,649 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:07,657 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/fbc47ad0637645a1b765b39b91b84361 2023-04-08 02:56:07,724 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:56:07,724 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:07,758 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:07,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 217 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922627748, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:07,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922627749, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,759 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:07,759 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922627750, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,760 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:07,760 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50238 deadline: 1680922627755, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,765 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:07,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922627759, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,864 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:07,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 219 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922627860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:07,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922627860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:07,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922627860, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,869 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:07,869 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50238 deadline: 1680922627861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:07,874 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:07,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922627867, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:08,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922628066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:08,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922628066, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,069 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:08,069 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 221 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922628067, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,073 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:08,074 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50238 deadline: 1680922628070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,074 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/b5391f2d561a4104a4cceea64c2f7a87 2023-04-08 02:56:08,079 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:08,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922628076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,372 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:08,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922628370, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,373 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:08,373 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 223 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922628371, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,376 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:08,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 222 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922628372, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,378 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:08,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50238 deadline: 1680922628375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,384 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:08,384 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922628382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,484 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=490 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/0643bf1e6aaa45b183155c692bedcefd 2023-04-08 02:56:08,490 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/fbc47ad0637645a1b765b39b91b84361 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/fbc47ad0637645a1b765b39b91b84361 2023-04-08 02:56:08,494 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/fbc47ad0637645a1b765b39b91b84361, entries=150, sequenceid=490, filesize=11.9 K 2023-04-08 02:56:08,495 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/b5391f2d561a4104a4cceea64c2f7a87 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/b5391f2d561a4104a4cceea64c2f7a87 2023-04-08 02:56:08,499 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/b5391f2d561a4104a4cceea64c2f7a87, entries=150, sequenceid=490, filesize=11.9 K 2023-04-08 02:56:08,500 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/0643bf1e6aaa45b183155c692bedcefd as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/0643bf1e6aaa45b183155c692bedcefd 2023-04-08 02:56:08,503 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/0643bf1e6aaa45b183155c692bedcefd, entries=150, sequenceid=490, filesize=11.9 K 2023-04-08 02:56:08,504 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=114.05 KB/116790 for 61e4c342830c067a5389bf73aa5cd767 in 856ms, sequenceid=490, compaction requested=true 2023-04-08 02:56:08,504 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:08,504 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:08,504 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:08,504 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:08,504 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,504 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:08,504 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:08,506 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,506 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:08,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:08,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:08,506 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:08,506 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:08,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:08,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:08,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:08,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:08,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:08,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@25ef2fb0[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:08,508 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:08,508 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:08,509 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:08,509 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:08,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:08,509 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:08,509 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:08,509 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:08,509 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,509 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:08,513 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:08,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:08,513 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:08,513 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:08,513 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:08,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:08,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:08,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:08,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:08,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:08,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:08,517 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:08,517 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:08,517 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:08,517 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:08,517 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:08,517 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:08,519 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:08,519 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:08,519 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:08,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:08,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:08,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:08,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:08,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:08,519 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,519 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:08,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:08,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:08,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:08,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:08,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:08,520 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:08,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:08,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:08,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,523 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,523 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:08,523 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:08,523 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:08,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:08,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:08,524 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:08,524 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:08,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:08,524 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:08,524 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:08,524 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:08,524 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,524 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:08,524 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:08,524 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:08,524 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:08,524 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:08,524 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:08,624 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:08,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:08,626 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:08,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:08,627 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:08,627 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:08,627 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:08,627 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:08,627 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:08,628 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:08,628 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:08,629 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:08,629 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,630 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,630 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:08,631 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:08,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:08,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:08,632 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:08,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:08,632 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:08,632 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:08,632 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:08,632 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:08,632 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:08,632 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:08,632 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:08,632 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:08,632 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:08,632 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:08,632 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:08,633 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=114.05 KB heapSize=339.42 KB 2023-04-08 02:56:08,633 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:56:08,633 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:08,633 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:56:08,633 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:08,633 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:56:08,633 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:08,641 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/09356be0f22f4590b7d7e3484d0d65da 2023-04-08 02:56:08,881 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:56:08,881 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:08,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:08,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922628889, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,896 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:08,897 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 226 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922628890, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,898 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:08,899 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 145 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50238 deadline: 1680922628895, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,901 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:08,901 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 183 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922628896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:08,903 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:08,903 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 227 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922628896, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,000 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:09,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922628997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,000 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:09,000 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922628997, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,003 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:09,004 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 147 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50238 deadline: 1680922628999, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,006 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:09,006 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 185 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922629002, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,007 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:09,007 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 229 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922629004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,054 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/9fc0f7b3489f4573b2d0fcc17accfff9 2023-04-08 02:56:09,208 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:09,208 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922629201, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:09,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922629202, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:09,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50238 deadline: 1680922629204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,213 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:09,213 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922629207, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,214 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:09,214 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 231 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922629209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,463 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=38.02 KB at sequenceid=510 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/897599af6d734f41aa72d2282717b50e 2023-04-08 02:56:09,467 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/09356be0f22f4590b7d7e3484d0d65da as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/09356be0f22f4590b7d7e3484d0d65da 2023-04-08 02:56:09,470 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/09356be0f22f4590b7d7e3484d0d65da, entries=150, sequenceid=510, filesize=11.9 K 2023-04-08 02:56:09,471 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/9fc0f7b3489f4573b2d0fcc17accfff9 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/9fc0f7b3489f4573b2d0fcc17accfff9 2023-04-08 02:56:09,481 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/9fc0f7b3489f4573b2d0fcc17accfff9, entries=150, sequenceid=510, filesize=11.9 K 2023-04-08 02:56:09,482 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/897599af6d734f41aa72d2282717b50e as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/897599af6d734f41aa72d2282717b50e 2023-04-08 02:56:09,486 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/897599af6d734f41aa72d2282717b50e, entries=150, sequenceid=510, filesize=11.9 K 2023-04-08 02:56:09,486 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~114.05 KB/116790, heapSize ~339.38 KB/347520, currentSize=67.09 KB/68700 for 61e4c342830c067a5389bf73aa5cd767 in 853ms, sequenceid=510, compaction requested=true 2023-04-08 02:56:09,486 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:09,487 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:09,487 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:09,487 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:09,487 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,487 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:09,487 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:09,488 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,489 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:09,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:09,489 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,489 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:09,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:09,489 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:09,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:09,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:09,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:09,492 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@532c8e85[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:09,492 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:09,492 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,493 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,493 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,493 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:09,493 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:09,493 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:09,494 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,494 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:09,496 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:09,496 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:09,496 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:09,496 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:09,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:09,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:09,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:09,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:09,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:09,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:09,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:09,500 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:09,500 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:09,501 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:09,501 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:09,501 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:09,501 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:09,502 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,502 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,503 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:09,503 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:09,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:09,504 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:09,504 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:09,505 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:09,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,510 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,510 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:09,510 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:09,511 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,511 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,511 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,511 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,511 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:09,512 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:09,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:09,512 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:09,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:09,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:09,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:09,512 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:09,512 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:09,512 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:09,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:09,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:09,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:56:09,518 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=73.80 KB heapSize=219.89 KB 2023-04-08 02:56:09,519 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:56:09,519 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:09,519 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:56:09,519 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:09,519 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:56:09,519 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:09,548 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:09,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50238 deadline: 1680922629535, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:09,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922629539, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:09,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 235 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922629541, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:09,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922629543, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,552 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:09,552 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922629548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,612 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:09,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:09,614 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:09,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:09,617 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:09,617 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:09,617 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:09,617 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:09,617 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:09,617 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,617 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:09,620 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:09,620 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:09,620 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:09,620 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:09,621 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,621 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:09,621 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:09,621 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:09,621 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:09,632 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:09,632 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:09,632 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:09,632 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:09,632 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:09,632 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:09,632 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:09,632 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:09,632 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:09,632 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:09,632 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:09,633 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:09,633 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:09,633 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,633 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:09,633 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:09,634 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,634 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:09,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:09,635 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,635 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:09,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:09,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:09,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:09,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:09,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:09,636 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@168529d9[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:09,636 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:09,636 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,637 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,637 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,638 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:09,638 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:09,638 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,638 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:09,638 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:09,639 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:09,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:09,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:09,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:09,640 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:09,640 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:09,640 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:09,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:09,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:09,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:09,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:09,641 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:09,641 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:09,641 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:09,641 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:09,642 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:09,642 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:09,644 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,644 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:09,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:09,644 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:09,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,644 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:09,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:09,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:09,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:09,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:09,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:09,645 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:09,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:09,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,647 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:09,647 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:09,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:09,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:09,648 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:09,647 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:09,648 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:09,647 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:09,648 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:09,648 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:09,648 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:09,648 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:09,648 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,648 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:09,648 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:09,648 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,648 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,648 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:09,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50238 deadline: 1680922629649, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:09,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922629651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,657 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:09,657 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 237 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922629651, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:09,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922629653, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,660 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:09,660 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922629654, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,748 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:09,749 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:09,752 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:09,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:09,755 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:09,755 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:09,755 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:09,755 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:09,755 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:09,756 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,756 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:09,757 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,757 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:09,757 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:09,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:09,758 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,758 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:09,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:09,758 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:09,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:09,759 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:09,760 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:09,760 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:09,760 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:09,760 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:09,760 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:09,760 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:09,760 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:09,760 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:09,761 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:09,761 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:09,761 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:09,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:09,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:09,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:09,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:09,762 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,762 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:09,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:09,763 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,763 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:09,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:09,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:09,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:09,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:09,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:09,765 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@9085c8f[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:09,765 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:09,765 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,769 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,769 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,769 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:09,769 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:09,769 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:09,769 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,769 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:09,770 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,770 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:09,771 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:09,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:09,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:09,771 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:09,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:09,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:09,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:09,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:09,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:09,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:09,774 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:09,774 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:09,774 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:09,774 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:09,774 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:09,774 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:09,776 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,776 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:09,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:09,776 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:09,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:09,776 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:09,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:09,776 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:09,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:09,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:09,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:09,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:09,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,778 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,780 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:09,780 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:09,780 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:09,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:09,781 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:09,781 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,781 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:09,781 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:09,781 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,781 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,781 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,781 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:09,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:09,781 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:09,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:09,781 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:09,781 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:09,866 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:09,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 158 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50238 deadline: 1680922629858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:09,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922629858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:09,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 239 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922629858, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,867 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:09,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922629861, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,868 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:09,868 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922629862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,881 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:09,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:09,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:09,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:09,888 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:09,888 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:09,888 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:09,888 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:09,888 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:09,888 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,888 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:09,891 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:09,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:09,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:09,891 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:09,892 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,892 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:09,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:09,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:09,892 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:09,893 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:09,893 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:09,893 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:09,893 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:09,893 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:09,893 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:09,893 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:09,893 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:09,893 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:09,893 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:09,893 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:09,893 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:09,894 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:09,894 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,894 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:09,894 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:09,895 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,895 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:09,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:09,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:09,896 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:09,897 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,897 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:09,897 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:09,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:09,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:09,899 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@55c660b1[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:09,899 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:09,899 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,900 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,900 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:09,900 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:09,900 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:09,900 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,900 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:09,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:09,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:09,902 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:09,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:09,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:09,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:09,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:09,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:09,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:09,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:09,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:09,904 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:09,904 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:09,905 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:09,905 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:09,905 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:09,905 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:09,907 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,907 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,907 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:09,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:09,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:09,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,908 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,908 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:09,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:09,908 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:09,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,910 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:09,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:09,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:09,912 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:09,912 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,912 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:09,912 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:09,912 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:09,912 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:09,912 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:09,912 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,912 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:09,912 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:09,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:09,912 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:09,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:09,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:09,912 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:09,913 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:09,913 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:09,913 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:09,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:09,913 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:09,943 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=525 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/1099dd08e93c44d398dfe6e1f3e2dacd 2023-04-08 02:56:10,013 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:10,013 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:10,015 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:10,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:10,016 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:10,016 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:10,016 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:10,016 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:10,016 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:10,016 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,016 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:10,022 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,022 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:10,023 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:10,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:10,023 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,023 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:10,023 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:10,023 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:10,024 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:10,024 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:10,024 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:10,024 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:10,024 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:10,024 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:10,024 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:10,024 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:10,024 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:10,024 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:10,024 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:10,024 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:10,024 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:10,024 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:10,025 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:10,025 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,025 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:10,025 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:10,027 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,027 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:10,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:10,027 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:10,028 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:10,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:10,028 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:10,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:10,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:10,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,028 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:10,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:10,029 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6229942a[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:10,029 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:10,029 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:10,030 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:10,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:10,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:10,030 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:10,031 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:10,031 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:10,031 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,031 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:10,032 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,032 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:10,032 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:10,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:10,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:10,032 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:10,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:10,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:10,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:10,033 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:10,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:10,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,034 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:10,034 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:10,034 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:10,035 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:10,035 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:10,035 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:10,035 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:10,036 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,036 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:10,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:10,036 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,036 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:10,037 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:10,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:10,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:10,037 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:10,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:10,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:10,038 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:10,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:10,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,041 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,041 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:10,041 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:10,041 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:10,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:10,041 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:10,041 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:10,041 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:10,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:10,041 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,041 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:10,041 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:10,041 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:10,041 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:10,042 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:10,042 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,042 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:10,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:10,141 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:10,142 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:10,144 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:10,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:10,145 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:10,145 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:10,145 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:10,145 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:10,145 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:10,145 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,145 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:10,147 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,147 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:10,147 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:10,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:10,147 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,147 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:10,147 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:10,147 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:10,148 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:10,148 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:10,148 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:10,148 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:10,148 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:10,148 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:10,148 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:10,148 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:10,148 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:10,148 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:10,148 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:10,148 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:10,149 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:10,149 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:10,149 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:10,149 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,149 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:10,149 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:10,151 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,151 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:10,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:10,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:10,151 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:10,151 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:10,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:10,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:10,151 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:10,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,152 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:10,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:10,152 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ddda257[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:10,152 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:10,152 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:10,153 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:10,154 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:10,154 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:10,154 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:10,154 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:10,154 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,154 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:10,154 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:10,156 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:10,156 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:10,156 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,156 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:10,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:10,156 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:10,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:10,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:10,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:10,157 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:10,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:10,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,158 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:10,158 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:10,158 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:10,159 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:10,159 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:10,159 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:10,159 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:10,160 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,160 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,160 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,160 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:10,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:10,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:10,161 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:10,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:10,161 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:10,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:10,161 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:10,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:10,162 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,162 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:10,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:10,163 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,165 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:10,165 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:10,165 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:10,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:10,165 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:10,165 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:10,165 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:10,165 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:10,165 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,165 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:10,165 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:10,165 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:10,165 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:10,165 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:10,165 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,165 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:10,165 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:10,166 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:10,173 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:10,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 160 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50238 deadline: 1680922630168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,173 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:10,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 198 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922630168, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,173 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:10,173 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922630169, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,174 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:10,175 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 241 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922630170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,178 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:10,178 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 243 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922630173, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,266 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:10,266 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:10,267 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:10,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:10,268 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:10,268 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:10,268 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:10,268 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:10,268 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:10,269 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,269 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:10,270 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:10,270 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,270 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:10,270 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:10,270 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,271 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:10,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:10,271 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:10,271 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:10,271 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:10,271 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:10,271 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:10,272 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:10,272 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:10,272 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:10,272 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:10,272 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:10,272 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:10,272 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:10,272 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:10,272 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:10,272 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:10,272 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:10,272 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,272 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:10,272 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:10,279 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,279 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:10,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:10,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:10,279 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:10,279 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:10,280 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:10,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:10,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:10,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,280 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:10,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:10,281 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@167a7147[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:10,281 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:10,281 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:10,282 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:10,282 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:10,282 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:10,282 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:10,282 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:10,282 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:10,282 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,283 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:10,284 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,284 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:10,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:10,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:10,284 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:10,284 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:10,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:10,284 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:10,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:10,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:10,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:10,285 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,286 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:10,286 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:10,286 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:10,286 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:10,286 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:10,286 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:10,286 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:10,288 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,288 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,288 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:10,288 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:10,289 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:10,289 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,290 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:10,290 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:10,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:10,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,291 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:10,291 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:10,292 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,294 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,294 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:10,294 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:10,294 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,294 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:10,294 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:10,294 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,294 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:10,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:10,294 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:10,294 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:10,294 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:10,294 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:10,294 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:10,294 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:10,294 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:10,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:10,295 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:10,295 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:10,351 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=525 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/b124de46a2da4927b0825759dc05b185 2023-04-08 02:56:10,368 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=525 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/2af39df1563a4ed6889b83686dc1bfe9 2023-04-08 02:56:10,373 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/1099dd08e93c44d398dfe6e1f3e2dacd as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/1099dd08e93c44d398dfe6e1f3e2dacd 2023-04-08 02:56:10,379 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/1099dd08e93c44d398dfe6e1f3e2dacd, entries=250, sequenceid=525, filesize=16.7 K 2023-04-08 02:56:10,380 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/b124de46a2da4927b0825759dc05b185 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/b124de46a2da4927b0825759dc05b185 2023-04-08 02:56:10,383 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/b124de46a2da4927b0825759dc05b185, entries=150, sequenceid=525, filesize=11.9 K 2023-04-08 02:56:10,385 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/2af39df1563a4ed6889b83686dc1bfe9 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/2af39df1563a4ed6889b83686dc1bfe9 2023-04-08 02:56:10,394 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:10,395 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:10,397 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:10,397 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/2af39df1563a4ed6889b83686dc1bfe9, entries=150, sequenceid=525, filesize=11.9 K 2023-04-08 02:56:10,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:10,398 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:10,398 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:10,398 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:10,398 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:10,398 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:10,398 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 61e4c342830c067a5389bf73aa5cd767 in 880ms, sequenceid=525, compaction requested=true 2023-04-08 02:56:10,398 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:10,398 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:10,398 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:10,398 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,398 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:10,398 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 02:56:10,398 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 7 store files, 0 compacting, 7 eligible, 16 blocking 2023-04-08 02:56:10,398 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:10,400 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:10,400 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:10,400 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:10,400 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:10,401 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,401 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:10,401 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:10,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:10,401 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:10,402 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:10,402 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:10,402 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:10,402 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:10,402 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:10,402 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:10,402 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:10,402 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:10,402 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:10,402 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:10,403 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62020 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 02:56:10,403 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/B is initiating minor compaction (all files) 2023-04-08 02:56:10,403 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/B in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:10,403 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/6f8af884c4484b1b83516f5f0cab1b86, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/07d9ddd005a34647b693f03943c13aa1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/b5391f2d561a4104a4cceea64c2f7a87, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/9fc0f7b3489f4573b2d0fcc17accfff9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/b124de46a2da4927b0825759dc05b185] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=60.6 K 2023-04-08 02:56:10,403 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-08 02:56:10,404 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:56:10,404 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:10,404 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:56:10,404 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:10,404 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:56:10,404 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:10,404 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 6f8af884c4484b1b83516f5f0cab1b86, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1680922566952 2023-04-08 02:56:10,405 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 7 files of size 98524 starting at candidate #0 after considering 15 permutations with 15 in ratio 2023-04-08 02:56:10,405 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 07d9ddd005a34647b693f03943c13aa1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1680922566973 2023-04-08 02:56:10,405 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/A is initiating minor compaction (all files) 2023-04-08 02:56:10,405 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/A in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:10,405 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/d3aa99f0dd904c3aa2f361276e674dbf, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/c77ecc95ca844a289bbb16fe7ba39096, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/004a4112001144159a6c0f713ba20c0b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/87fc3014923c46a495f0f1ce06c7903b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/fbc47ad0637645a1b765b39b91b84361, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/09356be0f22f4590b7d7e3484d0d65da, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/1099dd08e93c44d398dfe6e1f3e2dacd] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=96.2 K 2023-04-08 02:56:10,406 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting b5391f2d561a4104a4cceea64c2f7a87, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1680922567102 2023-04-08 02:56:10,406 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting d3aa99f0dd904c3aa2f361276e674dbf, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=424, earliestPutTs=1680922565172 2023-04-08 02:56:10,406 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 9fc0f7b3489f4573b2d0fcc17accfff9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1680922567735 2023-04-08 02:56:10,406 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting c77ecc95ca844a289bbb16fe7ba39096, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=443, earliestPutTs=1680922566318 2023-04-08 02:56:10,407 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting b124de46a2da4927b0825759dc05b185, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=525, earliestPutTs=1680922568888 2023-04-08 02:56:10,407 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 004a4112001144159a6c0f713ba20c0b, keycount=300, bloomtype=ROW, size=19.0 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1680922566948 2023-04-08 02:56:10,408 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 87fc3014923c46a495f0f1ce06c7903b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1680922566973 2023-04-08 02:56:10,416 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting fbc47ad0637645a1b765b39b91b84361, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1680922567102 2023-04-08 02:56:10,417 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 09356be0f22f4590b7d7e3484d0d65da, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1680922567735 2023-04-08 02:56:10,418 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 1099dd08e93c44d398dfe6e1f3e2dacd, keycount=250, bloomtype=ROW, size=16.7 K, encoding=NONE, compression=NONE, seqNum=525, earliestPutTs=1680922568888 2023-04-08 02:56:10,442 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#B#compaction#564 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:10,459 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#A#compaction#565 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:10,482 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/1c864b4f09784d7e8a74be6a36a765b3 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/1c864b4f09784d7e8a74be6a36a765b3 2023-04-08 02:56:10,486 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 7 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/A of 61e4c342830c067a5389bf73aa5cd767 into 1c864b4f09784d7e8a74be6a36a765b3(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:10,486 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:10,486 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/A, priority=9, startTime=1680922570398; duration=0sec 2023-04-08 02:56:10,486 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:10,486 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 02:56:10,487 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 62020 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 02:56:10,488 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 61e4c342830c067a5389bf73aa5cd767/C is initiating minor compaction (all files) 2023-04-08 02:56:10,488 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 61e4c342830c067a5389bf73aa5cd767/C in TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:10,488 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/8967d100ce7445049b6d1303e93567c7, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/f41318b61a154d7c86ac5621c95f5b54, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/0643bf1e6aaa45b183155c692bedcefd, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/897599af6d734f41aa72d2282717b50e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/2af39df1563a4ed6889b83686dc1bfe9] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp, totalSize=60.6 K 2023-04-08 02:56:10,488 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 8967d100ce7445049b6d1303e93567c7, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=455, earliestPutTs=1680922566952 2023-04-08 02:56:10,488 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting f41318b61a154d7c86ac5621c95f5b54, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=475, earliestPutTs=1680922566973 2023-04-08 02:56:10,489 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/4313567778cf4413a677233e9b56be02 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/4313567778cf4413a677233e9b56be02 2023-04-08 02:56:10,489 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 0643bf1e6aaa45b183155c692bedcefd, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=490, earliestPutTs=1680922567102 2023-04-08 02:56:10,489 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 897599af6d734f41aa72d2282717b50e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=510, earliestPutTs=1680922567735 2023-04-08 02:56:10,490 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 2af39df1563a4ed6889b83686dc1bfe9, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=525, earliestPutTs=1680922568888 2023-04-08 02:56:10,493 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/B of 61e4c342830c067a5389bf73aa5cd767 into 4313567778cf4413a677233e9b56be02(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:10,493 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:10,493 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/B, priority=11, startTime=1680922570398; duration=0sec 2023-04-08 02:56:10,494 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:10,500 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 61e4c342830c067a5389bf73aa5cd767#C#compaction#566 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:10,547 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/ce6a1d907ab542c18cd708bd0f0c4dc1 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/ce6a1d907ab542c18cd708bd0f0c4dc1 2023-04-08 02:56:10,552 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 61e4c342830c067a5389bf73aa5cd767/C of 61e4c342830c067a5389bf73aa5cd767 into ce6a1d907ab542c18cd708bd0f0c4dc1(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:10,552 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:10,552 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767., storeName=61e4c342830c067a5389bf73aa5cd767/C, priority=11, startTime=1680922570398; duration=0sec 2023-04-08 02:56:10,553 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:10,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:56:10,677 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:10,693 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:10,693 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:10,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922630687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50238 deadline: 1680922630688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,693 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:10,693 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922630688, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,694 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:10,694 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 246 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922630689, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,700 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:10,700 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 247 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922630693, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,796 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:10,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922630794, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,796 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:10,796 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922630795, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,797 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:10,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50238 deadline: 1680922630796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,797 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:10,797 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 248 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922630796, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,803 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:10,803 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 249 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922630801, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:10,842 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/00d3a15f1af34ab1bc998aa9eaa9fac1 2023-04-08 02:56:10,871 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/bc7d7406392445c49cce49461b272530 2023-04-08 02:56:11,002 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:11,002 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:11,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922630998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:11,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922630998, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:11,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:11,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50238 deadline: 1680922631000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:11,005 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:11,005 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 250 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922631000, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:11,010 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:11,010 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 251 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922631005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:11,307 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:11,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50252 deadline: 1680922631303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:11,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:11,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 208 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50216 deadline: 1680922631306, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:11,311 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:11,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50238 deadline: 1680922631307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:11,311 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:11,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 252 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50250 deadline: 1680922631307, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:11,317 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=542 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/c43161e6fa7b423ab1d16cfee1e6c9ce 2023-04-08 02:56:11,318 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:11,318 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 253 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:50232 deadline: 1680922631313, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:11,322 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/00d3a15f1af34ab1bc998aa9eaa9fac1 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/00d3a15f1af34ab1bc998aa9eaa9fac1 2023-04-08 02:56:11,326 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/00d3a15f1af34ab1bc998aa9eaa9fac1, entries=150, sequenceid=542, filesize=11.9 K 2023-04-08 02:56:11,327 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/bc7d7406392445c49cce49461b272530 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/bc7d7406392445c49cce49461b272530 2023-04-08 02:56:11,330 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/bc7d7406392445c49cce49461b272530, entries=150, sequenceid=542, filesize=11.9 K 2023-04-08 02:56:11,332 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/c43161e6fa7b423ab1d16cfee1e6c9ce as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/c43161e6fa7b423ab1d16cfee1e6c9ce 2023-04-08 02:56:11,341 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/c43161e6fa7b423ab1d16cfee1e6c9ce, entries=150, sequenceid=542, filesize=11.9 K 2023-04-08 02:56:11,342 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 61e4c342830c067a5389bf73aa5cd767 in 940ms, sequenceid=542, compaction requested=false 2023-04-08 02:56:11,342 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:11,342 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:11,342 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:11,342 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:11,342 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:11,342 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:11,342 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:11,344 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:11,344 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:11,344 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:11,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:11,345 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:11,345 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:11,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:11,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:11,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:11,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:11,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:11,348 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@70df48d1[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:11,348 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:11,348 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:11,357 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:11,357 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:11,357 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:11,357 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:11,358 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:11,358 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:11,358 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:11,358 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:11,359 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:11,359 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:11,359 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:11,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:11,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:11,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:11,359 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:11,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:11,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:11,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:11,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:11,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:11,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:11,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:11,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:11,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:11,363 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:11,363 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:11,363 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:11,363 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:11,363 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:11,365 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:11,365 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:11,365 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:11,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:11,365 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:11,365 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:11,365 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:11,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:11,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:11,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:11,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:11,368 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:11,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:11,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:11,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:11,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:11,371 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:11,372 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:11,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:11,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:11,372 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:11,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:11,372 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:11,372 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:11,372 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:11,372 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:11,372 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:11,372 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:11,372 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:11,372 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:11,372 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:11,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:11,372 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:11,372 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:11,372 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:11,472 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:11,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:11,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:11,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:11,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:11,476 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:11,476 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:11,476 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:11,476 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:11,476 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:11,476 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:11,478 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:11,478 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:11,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:11,478 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:11,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:11,479 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:11,479 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:11,479 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:11,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:11,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:11,479 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:11,480 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:11,480 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:11,480 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:11,480 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:11,480 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:11,492 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:11,492 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:11,492 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:11,492 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. started... 2023-04-08 02:56:11,492 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-08 02:56:11,492 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:56:11,493 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:11,493 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:56:11,493 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:11,493 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:56:11,493 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:11,541 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=560 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/2782c734021c4a6a97823190a24e8586 2023-04-08 02:56:11,742 DEBUG [Thread-3383] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x426cb82a to 127.0.0.1:64826 2023-04-08 02:56:11,742 DEBUG [Thread-3383] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:56:11,744 DEBUG [Thread-3387] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x211582f2 to 127.0.0.1:64826 2023-04-08 02:56:11,744 DEBUG [Thread-3387] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:56:11,744 DEBUG [Thread-3385] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5f5363d8 to 127.0.0.1:64826 2023-04-08 02:56:11,745 DEBUG [Thread-3385] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:56:11,746 DEBUG [Thread-3389] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x65816b6e to 127.0.0.1:64826 2023-04-08 02:56:11,746 DEBUG [Thread-3389] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:56:11,746 DEBUG [Thread-3381] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x67346c47 to 127.0.0.1:64826 2023-04-08 02:56:11,747 DEBUG [Thread-3381] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:56:11,812 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:56:11,812 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. as already flushing 2023-04-08 02:56:11,812 DEBUG [Thread-3370] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x501c5a88 to 127.0.0.1:64826 2023-04-08 02:56:11,812 DEBUG [Thread-3370] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:56:11,818 DEBUG [Thread-3376] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1c29757d to 127.0.0.1:64826 2023-04-08 02:56:11,818 DEBUG [Thread-3376] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:56:11,818 DEBUG [Thread-3372] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x27f2d997 to 127.0.0.1:64826 2023-04-08 02:56:11,818 DEBUG [Thread-3372] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:56:11,818 DEBUG [Thread-3378] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x06edd017 to 127.0.0.1:64826 2023-04-08 02:56:11,818 DEBUG [Thread-3378] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:56:11,821 DEBUG [Thread-3374] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x34ad4aa8 to 127.0.0.1:64826 2023-04-08 02:56:11,821 DEBUG [Thread-3374] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:56:11,978 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=560 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/bcb3c3fc391e4f1ab7e474c17c1501bc 2023-04-08 02:56:11,994 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=560 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/237f8d55c617499dbf2134715eec9f19 2023-04-08 02:56:11,997 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/2782c734021c4a6a97823190a24e8586 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/2782c734021c4a6a97823190a24e8586 2023-04-08 02:56:12,000 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/2782c734021c4a6a97823190a24e8586, entries=150, sequenceid=560, filesize=11.9 K 2023-04-08 02:56:12,000 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/bcb3c3fc391e4f1ab7e474c17c1501bc as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/bcb3c3fc391e4f1ab7e474c17c1501bc 2023-04-08 02:56:12,003 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/bcb3c3fc391e4f1ab7e474c17c1501bc, entries=150, sequenceid=560, filesize=11.9 K 2023-04-08 02:56:12,004 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/237f8d55c617499dbf2134715eec9f19 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/237f8d55c617499dbf2134715eec9f19 2023-04-08 02:56:12,006 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/237f8d55c617499dbf2134715eec9f19, entries=150, sequenceid=560, filesize=11.9 K 2023-04-08 02:56:12,007 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=33.54 KB/34350 for 61e4c342830c067a5389bf73aa5cd767 in 515ms, sequenceid=560, compaction requested=true 2023-04-08 02:56:12,007 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:12,007 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:12,007 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:12,007 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:12,007 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:12,007 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:12,007 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:12,008 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:12,008 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:12,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:12,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:12,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:12,009 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:12,009 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:12,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:12,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:12,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:12,009 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:12,010 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:12,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:12,010 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5ef91a24[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:12,010 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:12,010 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:12,011 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:12,011 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:12,011 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:12,011 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:12,011 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:12,011 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:12,011 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:12,011 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:12,013 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:12,013 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:12,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:12,013 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:12,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:12,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:12,013 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:12,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:12,013 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:12,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:12,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:12,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:12,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:12,014 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:12,015 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:12,015 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:12,015 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:12,015 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:12,015 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:12,015 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:12,015 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:12,017 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:12,017 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:12,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:12,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:12,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:12,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:12,017 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:12,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:12,017 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:12,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:12,017 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:12,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:12,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:12,017 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:12,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:12,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:12,018 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:12,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:12,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:12,018 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:12,019 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:12,020 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:12,020 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:12,020 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:12,020 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:12,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:12,020 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:12,020 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:12,020 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:12,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:12,020 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:12,020 DEBUG [Thread-3380] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:12,020 DEBUG [Thread-3380] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:12,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:12,020 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:12,020 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:12,020 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:12,020 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:12,020 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:12,020 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:12,120 DEBUG [Thread-3380] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:12,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:12,121 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-08 02:56:12,121 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 101 2023-04-08 02:56:12,121 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 64 2023-04-08 02:56:12,121 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 102 2023-04-08 02:56:12,121 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 75 2023-04-08 02:56:12,121 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 101 2023-04-08 02:56:12,121 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-08 02:56:12,121 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-08 02:56:12,121 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(402): scanned 2045 2023-04-08 02:56:12,121 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(403): verified 6127 rows 2023-04-08 02:56:12,121 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(402): scanned 2046 2023-04-08 02:56:12,121 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(403): verified 6125 rows 2023-04-08 02:56:12,122 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(402): scanned 2048 2023-04-08 02:56:12,122 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(403): verified 6143 rows 2023-04-08 02:56:12,122 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(402): scanned 2046 2023-04-08 02:56:12,122 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(403): verified 6138 rows 2023-04-08 02:56:12,122 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(402): scanned 2047 2023-04-08 02:56:12,122 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(403): verified 6141 rows 2023-04-08 02:56:12,122 INFO [Listener at localhost/39741] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-08 02:56:12,122 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x5e372785 to 127.0.0.1:64826 2023-04-08 02:56:12,122 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:56:12,123 INFO [Listener at localhost/39741] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-08 02:56:12,124 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-08 02:56:12,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure2.ProcedureExecutor(1029): Stored pid=50, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-08 02:56:12,127 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-08 02:56:12,127 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680922572127"}]},"ts":"1680922572127"} 2023-04-08 02:56:12,128 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-08 02:56:12,131 INFO [PEWorker-3] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-08 02:56:12,132 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=61e4c342830c067a5389bf73aa5cd767, UNASSIGN}] 2023-04-08 02:56:12,133 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=51, ppid=50, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=61e4c342830c067a5389bf73aa5cd767, UNASSIGN 2023-04-08 02:56:12,134 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=61e4c342830c067a5389bf73aa5cd767, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:12,134 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922572134"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680922572134"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680922572134"}]},"ts":"1680922572134"} 2023-04-08 02:56:12,135 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=52, ppid=51, state=RUNNABLE; CloseRegionProcedure 61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983}] 2023-04-08 02:56:12,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-08 02:56:12,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:56:12,286 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 61e4c342830c067a5389bf73aa5cd767, disabling compactions & flushes 2023-04-08 02:56:12,286 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:12,287 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:12,287 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. after waiting 0 ms 2023-04-08 02:56:12,287 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:12,287 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 61e4c342830c067a5389bf73aa5cd767 3/3 column families, dataSize=33.54 KB heapSize=100.36 KB 2023-04-08 02:56:12,287 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=A 2023-04-08 02:56:12,287 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:12,287 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=B 2023-04-08 02:56:12,287 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:12,287 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 61e4c342830c067a5389bf73aa5cd767, store=C 2023-04-08 02:56:12,287 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:12,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-08 02:56:12,695 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/92ab3d411fb24e3a86d3e3997d3fd6c9 2023-04-08 02:56:12,730 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-08 02:56:13,103 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/5da29c166307449b827f0718f4bffe22 2023-04-08 02:56:13,230 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-08 02:56:13,511 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=568 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/7611b1ff2ac94ba889a37638cdec98d9 2023-04-08 02:56:13,514 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/A/92ab3d411fb24e3a86d3e3997d3fd6c9 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/92ab3d411fb24e3a86d3e3997d3fd6c9 2023-04-08 02:56:13,518 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/92ab3d411fb24e3a86d3e3997d3fd6c9, entries=100, sequenceid=568, filesize=9.5 K 2023-04-08 02:56:13,519 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/B/5da29c166307449b827f0718f4bffe22 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/5da29c166307449b827f0718f4bffe22 2023-04-08 02:56:13,522 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/5da29c166307449b827f0718f4bffe22, entries=100, sequenceid=568, filesize=9.5 K 2023-04-08 02:56:13,523 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/.tmp/C/7611b1ff2ac94ba889a37638cdec98d9 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/7611b1ff2ac94ba889a37638cdec98d9 2023-04-08 02:56:13,527 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/7611b1ff2ac94ba889a37638cdec98d9, entries=100, sequenceid=568, filesize=9.5 K 2023-04-08 02:56:13,527 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~100.31 KB/102720, currentSize=0 B/0 for 61e4c342830c067a5389bf73aa5cd767 in 1240ms, sequenceid=568, compaction requested=true 2023-04-08 02:56:13,555 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/3499d35f07494df9a650a06206e42859, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/3e3c4c3896b34d77ad3c2f37885d406e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/5e411958352b4d8b9f732523078b84ee, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/6a42ee5ab47e4a91bf59c36e3cd55d85, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/241f943f8417466eb85ab943733ab161, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/93a402755947436b95f5f9760bc351a8, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/03f06a5c2a4949f6a33fbfff6ca77d8e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/b0ec3fd0501d49a2bc3bda12e58f0c46, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/dd41141929924724aede4688c6e45ee4, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/c49865cc4fa74b179b236adbc2e9eb02, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/432cca971e7645b28af7d8df906a1cd2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/4756e463510f42b0ab4c20bccb76d650, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/3e653d853970411ea5720da3395d8a2d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/5b1f8d639ebd448ca52e52cb8c518da9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/da00a91369824e3f912e2391b92877b6, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/440c79ecf73d46449bfbee86bdd9d776, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/2313e478712345f5a6a19734b8f63f34, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/aff25e5b612849c3abc53f12b9889f88, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/0adf181db4ed475b9b1dc2fa25d1d2d5, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/934fcc2b5702463a99bb3e32425a2741, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/ac64aff2506a49ff9352fabcbf3cf464, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/1ab86acd67e3408f868aed861ed47827, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/9acc26b83d444fe29f28a852539e50fe, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/2dcc4b54cac54f26b74dcaab2f2ea1e3, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/41eb067108cf4bb88b28f19d0f50b115, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/88606825bf214b25a7100a5b9b10d7d9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/88ef453cb6c445028435b771b1807763, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/446d87822356458b8afaba0ad7499add, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/d97e3390bb054efd910a418a48d11cf5, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/694bfa1098de42319dd25c89dbf90949, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/d3aa99f0dd904c3aa2f361276e674dbf, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/c77ecc95ca844a289bbb16fe7ba39096, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/004a4112001144159a6c0f713ba20c0b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/87fc3014923c46a495f0f1ce06c7903b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/fbc47ad0637645a1b765b39b91b84361, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/09356be0f22f4590b7d7e3484d0d65da, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/1099dd08e93c44d398dfe6e1f3e2dacd] to archive 2023-04-08 02:56:13,557 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 02:56:13,559 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/3499d35f07494df9a650a06206e42859 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/3499d35f07494df9a650a06206e42859 2023-04-08 02:56:13,560 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/3e3c4c3896b34d77ad3c2f37885d406e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/3e3c4c3896b34d77ad3c2f37885d406e 2023-04-08 02:56:13,561 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/5e411958352b4d8b9f732523078b84ee to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/5e411958352b4d8b9f732523078b84ee 2023-04-08 02:56:13,562 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/6a42ee5ab47e4a91bf59c36e3cd55d85 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/6a42ee5ab47e4a91bf59c36e3cd55d85 2023-04-08 02:56:13,563 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/241f943f8417466eb85ab943733ab161 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/241f943f8417466eb85ab943733ab161 2023-04-08 02:56:13,564 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/93a402755947436b95f5f9760bc351a8 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/93a402755947436b95f5f9760bc351a8 2023-04-08 02:56:13,565 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/03f06a5c2a4949f6a33fbfff6ca77d8e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/03f06a5c2a4949f6a33fbfff6ca77d8e 2023-04-08 02:56:13,566 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/b0ec3fd0501d49a2bc3bda12e58f0c46 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/b0ec3fd0501d49a2bc3bda12e58f0c46 2023-04-08 02:56:13,568 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/dd41141929924724aede4688c6e45ee4 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/dd41141929924724aede4688c6e45ee4 2023-04-08 02:56:13,569 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/c49865cc4fa74b179b236adbc2e9eb02 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/c49865cc4fa74b179b236adbc2e9eb02 2023-04-08 02:56:13,570 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/432cca971e7645b28af7d8df906a1cd2 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/432cca971e7645b28af7d8df906a1cd2 2023-04-08 02:56:13,571 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/4756e463510f42b0ab4c20bccb76d650 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/4756e463510f42b0ab4c20bccb76d650 2023-04-08 02:56:13,572 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/3e653d853970411ea5720da3395d8a2d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/3e653d853970411ea5720da3395d8a2d 2023-04-08 02:56:13,574 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/5b1f8d639ebd448ca52e52cb8c518da9 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/5b1f8d639ebd448ca52e52cb8c518da9 2023-04-08 02:56:13,575 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/da00a91369824e3f912e2391b92877b6 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/da00a91369824e3f912e2391b92877b6 2023-04-08 02:56:13,576 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/440c79ecf73d46449bfbee86bdd9d776 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/440c79ecf73d46449bfbee86bdd9d776 2023-04-08 02:56:13,577 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/2313e478712345f5a6a19734b8f63f34 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/2313e478712345f5a6a19734b8f63f34 2023-04-08 02:56:13,578 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/aff25e5b612849c3abc53f12b9889f88 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/aff25e5b612849c3abc53f12b9889f88 2023-04-08 02:56:13,579 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/0adf181db4ed475b9b1dc2fa25d1d2d5 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/0adf181db4ed475b9b1dc2fa25d1d2d5 2023-04-08 02:56:13,580 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/934fcc2b5702463a99bb3e32425a2741 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/934fcc2b5702463a99bb3e32425a2741 2023-04-08 02:56:13,581 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/ac64aff2506a49ff9352fabcbf3cf464 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/ac64aff2506a49ff9352fabcbf3cf464 2023-04-08 02:56:13,582 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/1ab86acd67e3408f868aed861ed47827 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/1ab86acd67e3408f868aed861ed47827 2023-04-08 02:56:13,583 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/9acc26b83d444fe29f28a852539e50fe to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/9acc26b83d444fe29f28a852539e50fe 2023-04-08 02:56:13,583 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/2dcc4b54cac54f26b74dcaab2f2ea1e3 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/2dcc4b54cac54f26b74dcaab2f2ea1e3 2023-04-08 02:56:13,584 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/41eb067108cf4bb88b28f19d0f50b115 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/41eb067108cf4bb88b28f19d0f50b115 2023-04-08 02:56:13,586 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/88606825bf214b25a7100a5b9b10d7d9 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/88606825bf214b25a7100a5b9b10d7d9 2023-04-08 02:56:13,587 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/88ef453cb6c445028435b771b1807763 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/88ef453cb6c445028435b771b1807763 2023-04-08 02:56:13,588 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/446d87822356458b8afaba0ad7499add to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/446d87822356458b8afaba0ad7499add 2023-04-08 02:56:13,589 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/d97e3390bb054efd910a418a48d11cf5 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/d97e3390bb054efd910a418a48d11cf5 2023-04-08 02:56:13,590 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/694bfa1098de42319dd25c89dbf90949 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/694bfa1098de42319dd25c89dbf90949 2023-04-08 02:56:13,591 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/d3aa99f0dd904c3aa2f361276e674dbf to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/d3aa99f0dd904c3aa2f361276e674dbf 2023-04-08 02:56:13,592 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/c77ecc95ca844a289bbb16fe7ba39096 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/c77ecc95ca844a289bbb16fe7ba39096 2023-04-08 02:56:13,593 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/004a4112001144159a6c0f713ba20c0b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/004a4112001144159a6c0f713ba20c0b 2023-04-08 02:56:13,593 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/87fc3014923c46a495f0f1ce06c7903b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/87fc3014923c46a495f0f1ce06c7903b 2023-04-08 02:56:13,594 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/fbc47ad0637645a1b765b39b91b84361 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/fbc47ad0637645a1b765b39b91b84361 2023-04-08 02:56:13,595 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/09356be0f22f4590b7d7e3484d0d65da to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/09356be0f22f4590b7d7e3484d0d65da 2023-04-08 02:56:13,596 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/1099dd08e93c44d398dfe6e1f3e2dacd to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/1099dd08e93c44d398dfe6e1f3e2dacd 2023-04-08 02:56:13,599 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/3b3d85d989e64d5a991469ce5ebe9e68, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/cf67290693414bb68a9c3b1125452f67, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/29ed95cd378d40ffb17a4dd671e1a5bb, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/f914c13f67f14beabbb64f26638a722e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/ac4a54b690fb40f48070ba4f01a3d689, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/ab4659ccb1a14712af3748beb99cae4a, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/44f1bb9bc5d749a5bbbe625f6b0d5ae9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/2fbb7de3332e4dd293ec407b243380ce, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/83c84fa075f943f3b7e6e6d9707b079d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/80d87c3dfd4c4b2ead0d5f078da49b8b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/c16abcf0e47f4c568e7ff0df740a291b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/eb4cbe428e0b4dd382eefc013ee7794e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/b578871f0ed14c84bb778dd12f59a1b2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/e13f05eb39cd443d92a2d1298680a410, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/7197bf88c4854ced8f6398cb10a5c21c, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/1bf655056dfc44318d6b48e356abecea, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/b23c166b2a644a79926a3193e9bebd12, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/712f7aa9f2db45fd9eccdc45367de9f7, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/8c895cc7bba349e98cd2f2013a82cfd6, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/f5f5aa780fee4142b149509eabbaa3a6, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/fa1e0045333b48a9948c5a7f55e973b4, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/c2b5687f151d4091b625909dbec1ab4e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/334a0f61eb854ee0b32c0c685479e80b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/0873b50d97de42fd98af374c071d40ab, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/fe65f867d34b4913abfa942f15654108, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/fb6f3f9f39914f56b2a0323bc6568d59, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/a753fc395b264beab8430cf5e2da7c32, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/ee5fb1e8201440c7a18bfc4761d332fd, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/63349a52891d4ae0afac176b2320d1c2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/fa7d34b48ffb4bcf9326d25c4fb7e81a, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/13b2d525ed8340e492d600e4d24be16e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/6f8af884c4484b1b83516f5f0cab1b86, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/ca9ed82efeeb42ed9de216ba6eb78a57, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/07d9ddd005a34647b693f03943c13aa1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/b5391f2d561a4104a4cceea64c2f7a87, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/9fc0f7b3489f4573b2d0fcc17accfff9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/b124de46a2da4927b0825759dc05b185] to archive 2023-04-08 02:56:13,600 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 02:56:13,601 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/3b3d85d989e64d5a991469ce5ebe9e68 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/3b3d85d989e64d5a991469ce5ebe9e68 2023-04-08 02:56:13,603 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/cf67290693414bb68a9c3b1125452f67 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/cf67290693414bb68a9c3b1125452f67 2023-04-08 02:56:13,603 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/29ed95cd378d40ffb17a4dd671e1a5bb to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/29ed95cd378d40ffb17a4dd671e1a5bb 2023-04-08 02:56:13,604 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/f914c13f67f14beabbb64f26638a722e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/f914c13f67f14beabbb64f26638a722e 2023-04-08 02:56:13,605 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/ac4a54b690fb40f48070ba4f01a3d689 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/ac4a54b690fb40f48070ba4f01a3d689 2023-04-08 02:56:13,606 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/ab4659ccb1a14712af3748beb99cae4a to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/ab4659ccb1a14712af3748beb99cae4a 2023-04-08 02:56:13,607 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/44f1bb9bc5d749a5bbbe625f6b0d5ae9 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/44f1bb9bc5d749a5bbbe625f6b0d5ae9 2023-04-08 02:56:13,608 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/2fbb7de3332e4dd293ec407b243380ce to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/2fbb7de3332e4dd293ec407b243380ce 2023-04-08 02:56:13,609 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/83c84fa075f943f3b7e6e6d9707b079d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/83c84fa075f943f3b7e6e6d9707b079d 2023-04-08 02:56:13,610 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/80d87c3dfd4c4b2ead0d5f078da49b8b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/80d87c3dfd4c4b2ead0d5f078da49b8b 2023-04-08 02:56:13,611 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/c16abcf0e47f4c568e7ff0df740a291b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/c16abcf0e47f4c568e7ff0df740a291b 2023-04-08 02:56:13,612 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/eb4cbe428e0b4dd382eefc013ee7794e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/eb4cbe428e0b4dd382eefc013ee7794e 2023-04-08 02:56:13,612 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/b578871f0ed14c84bb778dd12f59a1b2 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/b578871f0ed14c84bb778dd12f59a1b2 2023-04-08 02:56:13,613 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/e13f05eb39cd443d92a2d1298680a410 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/e13f05eb39cd443d92a2d1298680a410 2023-04-08 02:56:13,614 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/7197bf88c4854ced8f6398cb10a5c21c to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/7197bf88c4854ced8f6398cb10a5c21c 2023-04-08 02:56:13,616 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/1bf655056dfc44318d6b48e356abecea to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/1bf655056dfc44318d6b48e356abecea 2023-04-08 02:56:13,617 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/b23c166b2a644a79926a3193e9bebd12 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/b23c166b2a644a79926a3193e9bebd12 2023-04-08 02:56:13,618 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/712f7aa9f2db45fd9eccdc45367de9f7 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/712f7aa9f2db45fd9eccdc45367de9f7 2023-04-08 02:56:13,619 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/8c895cc7bba349e98cd2f2013a82cfd6 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/8c895cc7bba349e98cd2f2013a82cfd6 2023-04-08 02:56:13,620 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/f5f5aa780fee4142b149509eabbaa3a6 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/f5f5aa780fee4142b149509eabbaa3a6 2023-04-08 02:56:13,621 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/fa1e0045333b48a9948c5a7f55e973b4 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/fa1e0045333b48a9948c5a7f55e973b4 2023-04-08 02:56:13,623 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/c2b5687f151d4091b625909dbec1ab4e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/c2b5687f151d4091b625909dbec1ab4e 2023-04-08 02:56:13,624 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/334a0f61eb854ee0b32c0c685479e80b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/334a0f61eb854ee0b32c0c685479e80b 2023-04-08 02:56:13,625 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/0873b50d97de42fd98af374c071d40ab to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/0873b50d97de42fd98af374c071d40ab 2023-04-08 02:56:13,626 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/fe65f867d34b4913abfa942f15654108 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/fe65f867d34b4913abfa942f15654108 2023-04-08 02:56:13,627 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/fb6f3f9f39914f56b2a0323bc6568d59 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/fb6f3f9f39914f56b2a0323bc6568d59 2023-04-08 02:56:13,628 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/a753fc395b264beab8430cf5e2da7c32 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/a753fc395b264beab8430cf5e2da7c32 2023-04-08 02:56:13,629 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/ee5fb1e8201440c7a18bfc4761d332fd to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/ee5fb1e8201440c7a18bfc4761d332fd 2023-04-08 02:56:13,630 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/63349a52891d4ae0afac176b2320d1c2 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/63349a52891d4ae0afac176b2320d1c2 2023-04-08 02:56:13,631 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/fa7d34b48ffb4bcf9326d25c4fb7e81a to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/fa7d34b48ffb4bcf9326d25c4fb7e81a 2023-04-08 02:56:13,632 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/13b2d525ed8340e492d600e4d24be16e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/13b2d525ed8340e492d600e4d24be16e 2023-04-08 02:56:13,633 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/6f8af884c4484b1b83516f5f0cab1b86 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/6f8af884c4484b1b83516f5f0cab1b86 2023-04-08 02:56:13,634 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/ca9ed82efeeb42ed9de216ba6eb78a57 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/ca9ed82efeeb42ed9de216ba6eb78a57 2023-04-08 02:56:13,635 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/07d9ddd005a34647b693f03943c13aa1 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/07d9ddd005a34647b693f03943c13aa1 2023-04-08 02:56:13,636 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/b5391f2d561a4104a4cceea64c2f7a87 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/b5391f2d561a4104a4cceea64c2f7a87 2023-04-08 02:56:13,637 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/9fc0f7b3489f4573b2d0fcc17accfff9 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/9fc0f7b3489f4573b2d0fcc17accfff9 2023-04-08 02:56:13,638 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/b124de46a2da4927b0825759dc05b185 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/b124de46a2da4927b0825759dc05b185 2023-04-08 02:56:13,641 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/d184555db9c94742b15c4f68a6411216, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/98f8ccd9832344b9bf3188eef1098597, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/eb7ca69ac73a4d219ac10845578bf0e3, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/4fbb3e6acd104490ab3fa3d4c265cd25, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/28054d378aff4665832b7c46bfe3ffe1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/87c0cc3d7db743d8835402c8a5edefe0, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/31bad16a37854911909214542dc51809, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/4cb12aec980d4d05bc22657e5f62dda1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/beae83dc729e4325bd17f230aa6dea58, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/ef755e07c51b437a9ae99813825a8ca8, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/2572fdd8c62e44e5808adb015ec632e7, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/4369ebd17a6745c7963808cf038f470f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/4f114db4b039495c87c31be92a6d7fd9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/e8425a05f3d94f30a6af6b9abf0ccdc5, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/464b24b807634d5ca8c8f2a0ffd0d9a7, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/51e23a1b7a984d848a652342d252efa7, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/e8f73f8e37444d33aa79256b95582740, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/05fe345c4b3e49bfa7160bd3e07d6e11, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/c2cf393e27744e6ea53339de02dad836, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/299b0b60150f41e69e44767b98320346, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/8a1ce1acaec14d8cb7308cb4462fb534, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/b10816d244f24d81971994b53cf28ae7, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/c52f27e7527f46ec987375d1fd144d09, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/f1e2080c659c4719a448fffe5665cd89, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/b6815f1d6c884fb38b1d3a6c08af233c, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/e1116fb2355b449183d931c3806042af, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/bf3322389c0e4b39abdff279f49a6bc9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/effdd6c6427d46629022ab42084b45e5, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/5e6ea109808e48b3a778a6587d5dce3b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/032f25bdab4c4519b0e460234e736fd0, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/5c9b29e1d2174378b5f43c4988eca0dd, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/8967d100ce7445049b6d1303e93567c7, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/38bbd683d46e42b7adf51c3cf05c8514, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/f41318b61a154d7c86ac5621c95f5b54, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/0643bf1e6aaa45b183155c692bedcefd, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/897599af6d734f41aa72d2282717b50e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/2af39df1563a4ed6889b83686dc1bfe9] to archive 2023-04-08 02:56:13,642 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 02:56:13,643 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/d184555db9c94742b15c4f68a6411216 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/d184555db9c94742b15c4f68a6411216 2023-04-08 02:56:13,644 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/98f8ccd9832344b9bf3188eef1098597 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/98f8ccd9832344b9bf3188eef1098597 2023-04-08 02:56:13,645 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/eb7ca69ac73a4d219ac10845578bf0e3 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/eb7ca69ac73a4d219ac10845578bf0e3 2023-04-08 02:56:13,646 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/4fbb3e6acd104490ab3fa3d4c265cd25 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/4fbb3e6acd104490ab3fa3d4c265cd25 2023-04-08 02:56:13,647 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/28054d378aff4665832b7c46bfe3ffe1 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/28054d378aff4665832b7c46bfe3ffe1 2023-04-08 02:56:13,647 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/87c0cc3d7db743d8835402c8a5edefe0 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/87c0cc3d7db743d8835402c8a5edefe0 2023-04-08 02:56:13,649 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/31bad16a37854911909214542dc51809 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/31bad16a37854911909214542dc51809 2023-04-08 02:56:13,650 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/4cb12aec980d4d05bc22657e5f62dda1 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/4cb12aec980d4d05bc22657e5f62dda1 2023-04-08 02:56:13,651 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/beae83dc729e4325bd17f230aa6dea58 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/beae83dc729e4325bd17f230aa6dea58 2023-04-08 02:56:13,652 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/ef755e07c51b437a9ae99813825a8ca8 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/ef755e07c51b437a9ae99813825a8ca8 2023-04-08 02:56:13,653 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/2572fdd8c62e44e5808adb015ec632e7 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/2572fdd8c62e44e5808adb015ec632e7 2023-04-08 02:56:13,654 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/4369ebd17a6745c7963808cf038f470f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/4369ebd17a6745c7963808cf038f470f 2023-04-08 02:56:13,654 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/4f114db4b039495c87c31be92a6d7fd9 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/4f114db4b039495c87c31be92a6d7fd9 2023-04-08 02:56:13,655 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/e8425a05f3d94f30a6af6b9abf0ccdc5 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/e8425a05f3d94f30a6af6b9abf0ccdc5 2023-04-08 02:56:13,656 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/464b24b807634d5ca8c8f2a0ffd0d9a7 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/464b24b807634d5ca8c8f2a0ffd0d9a7 2023-04-08 02:56:13,656 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/51e23a1b7a984d848a652342d252efa7 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/51e23a1b7a984d848a652342d252efa7 2023-04-08 02:56:13,657 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/e8f73f8e37444d33aa79256b95582740 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/e8f73f8e37444d33aa79256b95582740 2023-04-08 02:56:13,658 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/05fe345c4b3e49bfa7160bd3e07d6e11 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/05fe345c4b3e49bfa7160bd3e07d6e11 2023-04-08 02:56:13,659 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/c2cf393e27744e6ea53339de02dad836 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/c2cf393e27744e6ea53339de02dad836 2023-04-08 02:56:13,660 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/299b0b60150f41e69e44767b98320346 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/299b0b60150f41e69e44767b98320346 2023-04-08 02:56:13,661 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/8a1ce1acaec14d8cb7308cb4462fb534 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/8a1ce1acaec14d8cb7308cb4462fb534 2023-04-08 02:56:13,661 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/b10816d244f24d81971994b53cf28ae7 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/b10816d244f24d81971994b53cf28ae7 2023-04-08 02:56:13,662 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/c52f27e7527f46ec987375d1fd144d09 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/c52f27e7527f46ec987375d1fd144d09 2023-04-08 02:56:13,663 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/f1e2080c659c4719a448fffe5665cd89 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/f1e2080c659c4719a448fffe5665cd89 2023-04-08 02:56:13,664 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/b6815f1d6c884fb38b1d3a6c08af233c to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/b6815f1d6c884fb38b1d3a6c08af233c 2023-04-08 02:56:13,664 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/e1116fb2355b449183d931c3806042af to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/e1116fb2355b449183d931c3806042af 2023-04-08 02:56:13,665 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/bf3322389c0e4b39abdff279f49a6bc9 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/bf3322389c0e4b39abdff279f49a6bc9 2023-04-08 02:56:13,666 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/effdd6c6427d46629022ab42084b45e5 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/effdd6c6427d46629022ab42084b45e5 2023-04-08 02:56:13,666 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/5e6ea109808e48b3a778a6587d5dce3b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/5e6ea109808e48b3a778a6587d5dce3b 2023-04-08 02:56:13,667 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/032f25bdab4c4519b0e460234e736fd0 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/032f25bdab4c4519b0e460234e736fd0 2023-04-08 02:56:13,668 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/5c9b29e1d2174378b5f43c4988eca0dd to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/5c9b29e1d2174378b5f43c4988eca0dd 2023-04-08 02:56:13,668 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/8967d100ce7445049b6d1303e93567c7 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/8967d100ce7445049b6d1303e93567c7 2023-04-08 02:56:13,669 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/38bbd683d46e42b7adf51c3cf05c8514 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/38bbd683d46e42b7adf51c3cf05c8514 2023-04-08 02:56:13,670 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/f41318b61a154d7c86ac5621c95f5b54 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/f41318b61a154d7c86ac5621c95f5b54 2023-04-08 02:56:13,671 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/0643bf1e6aaa45b183155c692bedcefd to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/0643bf1e6aaa45b183155c692bedcefd 2023-04-08 02:56:13,671 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/897599af6d734f41aa72d2282717b50e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/897599af6d734f41aa72d2282717b50e 2023-04-08 02:56:13,672 DEBUG [StoreCloser-TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/2af39df1563a4ed6889b83686dc1bfe9 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/2af39df1563a4ed6889b83686dc1bfe9 2023-04-08 02:56:13,676 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/recovered.edits/571.seqid, newMaxSeqId=571, maxSeqId=1 2023-04-08 02:56:13,677 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767. 2023-04-08 02:56:13,677 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 61e4c342830c067a5389bf73aa5cd767: 2023-04-08 02:56:13,679 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:56:13,679 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=51 updating hbase:meta row=61e4c342830c067a5389bf73aa5cd767, regionState=CLOSED 2023-04-08 02:56:13,679 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922573679"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680922573679"}]},"ts":"1680922573679"} 2023-04-08 02:56:13,681 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=52, resume processing ppid=51 2023-04-08 02:56:13,681 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=52, ppid=51, state=SUCCESS; CloseRegionProcedure 61e4c342830c067a5389bf73aa5cd767, server=jenkins-hbase4.apache.org,35089,1680922417983 in 1.5460 sec 2023-04-08 02:56:13,682 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=51, resume processing ppid=50 2023-04-08 02:56:13,682 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=51, ppid=50, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=61e4c342830c067a5389bf73aa5cd767, UNASSIGN in 1.5490 sec 2023-04-08 02:56:13,682 DEBUG [PEWorker-3] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680922573682"}]},"ts":"1680922573682"} 2023-04-08 02:56:13,683 INFO [PEWorker-3] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-08 02:56:13,684 INFO [PEWorker-3] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-08 02:56:13,685 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=50, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.5610 sec 2023-04-08 02:56:14,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=50 2023-04-08 02:56:14,231 INFO [Listener at localhost/39741] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 50 completed 2023-04-08 02:56:14,232 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-08 02:56:14,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure2.ProcedureExecutor(1029): Stored pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:56:14,234 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=53, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:56:14,234 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-08 02:56:14,234 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=53, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:56:14,238 DEBUG [HFileArchiver-11] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:56:14,240 DEBUG [HFileArchiver-11] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A, FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B, FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C, FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/recovered.edits] 2023-04-08 02:56:14,242 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/00d3a15f1af34ab1bc998aa9eaa9fac1 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/00d3a15f1af34ab1bc998aa9eaa9fac1 2023-04-08 02:56:14,243 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/1c864b4f09784d7e8a74be6a36a765b3 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/1c864b4f09784d7e8a74be6a36a765b3 2023-04-08 02:56:14,244 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/2782c734021c4a6a97823190a24e8586 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/2782c734021c4a6a97823190a24e8586 2023-04-08 02:56:14,245 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/92ab3d411fb24e3a86d3e3997d3fd6c9 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/A/92ab3d411fb24e3a86d3e3997d3fd6c9 2023-04-08 02:56:14,247 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/4313567778cf4413a677233e9b56be02 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/4313567778cf4413a677233e9b56be02 2023-04-08 02:56:14,248 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/5da29c166307449b827f0718f4bffe22 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/5da29c166307449b827f0718f4bffe22 2023-04-08 02:56:14,249 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/bc7d7406392445c49cce49461b272530 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/bc7d7406392445c49cce49461b272530 2023-04-08 02:56:14,250 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/bcb3c3fc391e4f1ab7e474c17c1501bc to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/B/bcb3c3fc391e4f1ab7e474c17c1501bc 2023-04-08 02:56:14,252 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/237f8d55c617499dbf2134715eec9f19 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/237f8d55c617499dbf2134715eec9f19 2023-04-08 02:56:14,253 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/7611b1ff2ac94ba889a37638cdec98d9 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/7611b1ff2ac94ba889a37638cdec98d9 2023-04-08 02:56:14,254 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/c43161e6fa7b423ab1d16cfee1e6c9ce to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/c43161e6fa7b423ab1d16cfee1e6c9ce 2023-04-08 02:56:14,255 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/ce6a1d907ab542c18cd708bd0f0c4dc1 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/C/ce6a1d907ab542c18cd708bd0f0c4dc1 2023-04-08 02:56:14,257 DEBUG [HFileArchiver-11] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/recovered.edits/571.seqid to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767/recovered.edits/571.seqid 2023-04-08 02:56:14,259 DEBUG [HFileArchiver-11] backup.HFileArchiver(596): Deleted hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/61e4c342830c067a5389bf73aa5cd767 2023-04-08 02:56:14,259 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-08 02:56:14,261 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=53, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:56:14,262 WARN [PEWorker-5] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-08 02:56:14,263 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-08 02:56:14,264 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=53, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:56:14,264 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-08 02:56:14,264 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680922574264"}]},"ts":"9223372036854775807"} 2023-04-08 02:56:14,265 INFO [PEWorker-5] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-08 02:56:14,265 DEBUG [PEWorker-5] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 61e4c342830c067a5389bf73aa5cd767, NAME => 'TestAcidGuarantees,,1680922549563.61e4c342830c067a5389bf73aa5cd767.', STARTKEY => '', ENDKEY => ''}] 2023-04-08 02:56:14,265 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-08 02:56:14,265 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680922574265"}]},"ts":"9223372036854775807"} 2023-04-08 02:56:14,266 INFO [PEWorker-5] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-08 02:56:14,268 DEBUG [PEWorker-5] procedure.DeleteTableProcedure(130): Finished pid=53, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:56:14,269 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=53, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 36 msec 2023-04-08 02:56:14,335 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=53 2023-04-08 02:56:14,335 INFO [Listener at localhost/39741] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 53 completed 2023-04-08 02:56:14,347 INFO [Listener at localhost/39741] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testScanAtomicity Thread=232 (was 227) - Thread LEAK? -, OpenFileDescriptor=577 (was 517) - OpenFileDescriptor LEAK? -, MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=424 (was 316) - SystemLoadAverage LEAK? -, ProcessCount=172 (was 172), AvailableMemoryMB=1821 (was 1372) - AvailableMemoryMB LEAK? - 2023-04-08 02:56:14,358 INFO [Listener at localhost/39741] hbase.ResourceChecker(147): before: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=232, OpenFileDescriptor=577, MaxFileDescriptor=60000, SystemLoadAverage=424, ProcessCount=172, AvailableMemoryMB=1821 2023-04-08 02:56:14,360 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-08 02:56:14,360 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.HMaster$4(2112): Client=jenkins//172.31.14.131 create 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-08 02:56:14,361 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure2.ProcedureExecutor(1029): Stored pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION; CreateTableProcedure table=TestAcidGuarantees 2023-04-08 02:56:14,362 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_PRE_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_PRE_OPERATION 2023-04-08 02:56:14,362 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(697): Client=jenkins//172.31.14.131 procedure request for creating table: namespace: "default" qualifier: "TestAcidGuarantees" procId is: 54 2023-04-08 02:56:14,363 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_WRITE_FS_LAYOUT, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_WRITE_FS_LAYOUT 2023-04-08 02:56:14,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-08 02:56:14,364 DEBUG [HFileArchiver-12] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:14,364 DEBUG [HFileArchiver-12] backup.HFileArchiver(153): Directory hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b empty. 2023-04-08 02:56:14,365 DEBUG [HFileArchiver-12] backup.HFileArchiver(599): Failed to delete directory hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:14,365 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-08 02:56:14,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-08 02:56:14,665 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-08 02:56:14,771 DEBUG [PEWorker-1] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-08 02:56:14,773 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(7675): creating {ENCODED => 337c881227bdfa873bb568857a6d7e4b, NAME => 'TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.', STARTKEY => '', ENDKEY => ''}, tableDescriptor='TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, regionDir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp 2023-04-08 02:56:14,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-08 02:56:15,178 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 02:56:15,179 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1604): Closing 337c881227bdfa873bb568857a6d7e4b, disabling compactions & flushes 2023-04-08 02:56:15,179 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:15,179 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:15,179 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. after waiting 0 ms 2023-04-08 02:56:15,179 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:15,179 INFO [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:15,179 DEBUG [RegionOpenAndInit-TestAcidGuarantees-pool-0] regionserver.HRegion(1558): Region close journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:15,181 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ADD_TO_META, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ADD_TO_META 2023-04-08 02:56:15,182 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922575181"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680922575181"}]},"ts":"1680922575181"} 2023-04-08 02:56:15,183 INFO [PEWorker-1] hbase.MetaTableAccessor(1496): Added 1 regions to meta. 2023-04-08 02:56:15,183 INFO [PEWorker-1] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_ASSIGN_REGIONS, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_ASSIGN_REGIONS 2023-04-08 02:56:15,183 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680922575183"}]},"ts":"1680922575183"} 2023-04-08 02:56:15,184 INFO [PEWorker-1] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLING in hbase:meta 2023-04-08 02:56:15,187 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=337c881227bdfa873bb568857a6d7e4b, ASSIGN}] 2023-04-08 02:56:15,188 INFO [PEWorker-4] procedure.MasterProcedureScheduler(727): Took xlock for pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE; TransitRegionStateProcedure table=TestAcidGuarantees, region=337c881227bdfa873bb568857a6d7e4b, ASSIGN 2023-04-08 02:56:15,188 INFO [PEWorker-4] assignment.TransitRegionStateProcedure(193): Starting pid=55, ppid=54, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=337c881227bdfa873bb568857a6d7e4b, ASSIGN; state=OFFLINE, location=jenkins-hbase4.apache.org,35089,1680922417983; forceNewPlan=false, retain=false 2023-04-08 02:56:15,340 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=337c881227bdfa873bb568857a6d7e4b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:15,340 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922575340"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680922575340"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680922575340"}]},"ts":"1680922575340"} 2023-04-08 02:56:15,341 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=56, ppid=55, state=RUNNABLE; OpenRegionProcedure 337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983}] 2023-04-08 02:56:15,466 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-08 02:56:15,495 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:15,495 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 337c881227bdfa873bb568857a6d7e4b, NAME => 'TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.', STARTKEY => '', ENDKEY => ''} 2023-04-08 02:56:15,496 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:15,496 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 02:56:15,496 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:15,496 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:15,497 INFO [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:15,498 DEBUG [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A 2023-04-08 02:56:15,498 DEBUG [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A 2023-04-08 02:56:15,499 INFO [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 02:56:15,499 INFO [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 337c881227bdfa873bb568857a6d7e4b columnFamilyName A 2023-04-08 02:56:15,499 INFO [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] regionserver.HStore(310): Store=337c881227bdfa873bb568857a6d7e4b/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:56:15,499 INFO [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:15,500 DEBUG [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B 2023-04-08 02:56:15,500 DEBUG [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B 2023-04-08 02:56:15,501 INFO [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 02:56:15,501 INFO [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 337c881227bdfa873bb568857a6d7e4b columnFamilyName B 2023-04-08 02:56:15,501 INFO [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] regionserver.HStore(310): Store=337c881227bdfa873bb568857a6d7e4b/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:56:15,501 INFO [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:15,502 DEBUG [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C 2023-04-08 02:56:15,503 DEBUG [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C 2023-04-08 02:56:15,503 INFO [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 02:56:15,503 INFO [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 337c881227bdfa873bb568857a6d7e4b columnFamilyName C 2023-04-08 02:56:15,503 INFO [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] regionserver.HStore(310): Store=337c881227bdfa873bb568857a6d7e4b/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:56:15,503 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:15,504 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:15,504 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:15,506 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-08 02:56:15,507 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:15,508 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/recovered.edits/1.seqid, newMaxSeqId=1, maxSeqId=-1 2023-04-08 02:56:15,509 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 337c881227bdfa873bb568857a6d7e4b; next sequenceid=2; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=64198903, jitterRate=-0.04336179792881012}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-08 02:56:15,509 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:15,509 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., pid=56, masterSystemTime=1680922575492 2023-04-08 02:56:15,511 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:15,511 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:15,511 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=55 updating hbase:meta row=337c881227bdfa873bb568857a6d7e4b, regionState=OPEN, openSeqNum=2, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:15,511 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922575511"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680922575511"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680922575511"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680922575511"}]},"ts":"1680922575511"} 2023-04-08 02:56:15,513 INFO [PEWorker-5] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=56, resume processing ppid=55 2023-04-08 02:56:15,513 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=56, ppid=55, state=SUCCESS; OpenRegionProcedure 337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 in 171 msec 2023-04-08 02:56:15,514 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=55, resume processing ppid=54 2023-04-08 02:56:15,514 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=55, ppid=54, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=337c881227bdfa873bb568857a6d7e4b, ASSIGN in 326 msec 2023-04-08 02:56:15,515 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_UPDATE_DESC_CACHE, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_UPDATE_DESC_CACHE 2023-04-08 02:56:15,515 DEBUG [PEWorker-4] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680922575515"}]},"ts":"1680922575515"} 2023-04-08 02:56:15,516 INFO [PEWorker-4] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=ENABLED in hbase:meta 2023-04-08 02:56:15,517 INFO [PEWorker-4] procedure.CreateTableProcedure(80): pid=54, state=RUNNABLE:CREATE_TABLE_POST_OPERATION, locked=true; CreateTableProcedure table=TestAcidGuarantees execute state=CREATE_TABLE_POST_OPERATION 2023-04-08 02:56:15,518 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=54, state=SUCCESS; CreateTableProcedure table=TestAcidGuarantees in 1.1580 sec 2023-04-08 02:56:16,467 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=54 2023-04-08 02:56:16,468 INFO [Listener at localhost/39741] client.HBaseAdmin$TableFuture(3541): Operation: CREATE, Table Name: default:TestAcidGuarantees, procId: 54 completed 2023-04-08 02:56:16,468 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x45878790 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:56:16,480 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@1e3503b4, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:56:16,481 DEBUG [Listener at localhost/39741] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:56:16,482 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46568, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:56:16,483 DEBUG [Listener at localhost/39741] ipc.RpcConnection(124): Using SIMPLE authentication for service=MasterService, sasl=false 2023-04-08 02:56:16,486 INFO [RS-EventLoopGroup-1-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46350, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=MasterService 2023-04-08 02:56:16,487 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] util.TableDescriptorChecker(340): MEMSTORE_FLUSHSIZE for table descriptor or "hbase.hregion.memstore.flush.size" (131072) is too small, which might cause very frequent flushing. 2023-04-08 02:56:16,487 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.HMaster$12(2454): Client=jenkins//172.31.14.131 modify table TestAcidGuarantees from 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} to 'TestAcidGuarantees', {TABLE_ATTRIBUTES => {METADATA => {'hbase.hregion.compacting.memstore.type' => 'EAGER', 'hbase.hregion.memstore.mslab.enabled' => 'false', 'hbase.memstore.inmemoryflush.threshold.factor' => '0.9'}}}, {NAME => 'A', MOB_THRESHOLD => '4', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', REPLICATION_SCOPE => '0', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', IS_MOB => 'true', COMPRESSION => 'NONE', BLOCKCACHE => 'true', BLOCKSIZE => '65536'}, {NAME => 'B', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'}, {NAME => 'C', BLOOMFILTER => 'ROW', IN_MEMORY => 'false', VERSIONS => '1', KEEP_DELETED_CELLS => 'FALSE', DATA_BLOCK_ENCODING => 'NONE', COMPRESSION => 'NONE', TTL => 'FOREVER', MIN_VERSIONS => '0', BLOCKCACHE => 'true', BLOCKSIZE => '65536', REPLICATION_SCOPE => '0'} 2023-04-08 02:56:16,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure2.ProcedureExecutor(1029): Stored pid=57, state=RUNNABLE:MODIFY_TABLE_PREPARE; ModifyTableProcedure table=TestAcidGuarantees 2023-04-08 02:56:16,500 DEBUG [PEWorker-2] util.FSTableDescriptors(570): Wrote into hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-08 02:56:16,501 DEBUG [PEWorker-2] util.FSTableDescriptors(519): Deleted hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000001 2023-04-08 02:56:16,501 INFO [PEWorker-2] util.FSTableDescriptors(293): Updated tableinfo=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/.tabledesc/.tableinfo.0000000002 2023-04-08 02:56:16,503 INFO [PEWorker-2] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=58, ppid=57, state=RUNNABLE:REOPEN_TABLE_REGIONS_GET_REGIONS; ReopenTableRegionsProcedure table=TestAcidGuarantees}] 2023-04-08 02:56:16,506 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=337c881227bdfa873bb568857a6d7e4b, REOPEN/MOVE}] 2023-04-08 02:56:16,507 INFO [PEWorker-5] procedure.MasterProcedureScheduler(727): Took xlock for pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=337c881227bdfa873bb568857a6d7e4b, REOPEN/MOVE 2023-04-08 02:56:16,508 INFO [PEWorker-5] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=337c881227bdfa873bb568857a6d7e4b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:16,508 DEBUG [PEWorker-5] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922576508"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680922576508"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680922576508"}]},"ts":"1680922576508"} 2023-04-08 02:56:16,509 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=60, ppid=59, state=RUNNABLE; CloseRegionProcedure 337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983}] 2023-04-08 02:56:16,661 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:16,662 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 337c881227bdfa873bb568857a6d7e4b, disabling compactions & flushes 2023-04-08 02:56:16,662 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:16,662 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:16,662 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. after waiting 0 ms 2023-04-08 02:56:16,662 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:16,666 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/recovered.edits/4.seqid, newMaxSeqId=4, maxSeqId=1 2023-04-08 02:56:16,666 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:16,666 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:16,666 WARN [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(3509): Not adding moved region record: 337c881227bdfa873bb568857a6d7e4b to self. 2023-04-08 02:56:16,668 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:16,668 INFO [PEWorker-4] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=337c881227bdfa873bb568857a6d7e4b, regionState=CLOSED 2023-04-08 02:56:16,668 DEBUG [PEWorker-4] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922576668"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680922576668"}]},"ts":"1680922576668"} 2023-04-08 02:56:16,671 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=60, resume processing ppid=59 2023-04-08 02:56:16,671 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=60, ppid=59, state=SUCCESS; CloseRegionProcedure 337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 in 160 msec 2023-04-08 02:56:16,671 INFO [PEWorker-2] assignment.TransitRegionStateProcedure(193): Starting pid=59, ppid=58, state=RUNNABLE:REGION_STATE_TRANSITION_GET_ASSIGN_CANDIDATE, locked=true; TransitRegionStateProcedure table=TestAcidGuarantees, region=337c881227bdfa873bb568857a6d7e4b, REOPEN/MOVE; state=CLOSED, location=jenkins-hbase4.apache.org,35089,1680922417983; forceNewPlan=false, retain=true 2023-04-08 02:56:16,822 INFO [PEWorker-3] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=337c881227bdfa873bb568857a6d7e4b, regionState=OPENING, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:16,822 DEBUG [PEWorker-3] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922576822"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680922576822"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680922576822"}]},"ts":"1680922576822"} 2023-04-08 02:56:16,823 INFO [PEWorker-3] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=61, ppid=59, state=RUNNABLE; OpenRegionProcedure 337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983}] 2023-04-08 02:56:16,977 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(128): Open TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:16,977 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7854): Opening region: {ENCODED => 337c881227bdfa873bb568857a6d7e4b, NAME => 'TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.', STARTKEY => '', ENDKEY => ''} 2023-04-08 02:56:16,978 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.MetricsRegionSourceImpl(79): Creating new MetricsRegionSourceImpl for table TestAcidGuarantees 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:16,978 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(866): Instantiated TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.; StoreHotnessProtector, parallelPutToStoreThreadLimit=10 ; minColumnNum=100 ; preparePutThreadLimit=20 ; hotProtect now enable 2023-04-08 02:56:16,978 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7894): checking encryption for 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:16,978 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(7897): checking classloading for 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:16,979 INFO [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family A of region 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:16,980 DEBUG [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A 2023-04-08 02:56:16,980 DEBUG [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A 2023-04-08 02:56:16,980 INFO [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] regionserver.CompactingMemStore(121): Store=A, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 02:56:16,980 INFO [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 337c881227bdfa873bb568857a6d7e4b columnFamilyName A 2023-04-08 02:56:16,981 INFO [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] regionserver.HStore(310): Store=337c881227bdfa873bb568857a6d7e4b/A, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:56:16,981 INFO [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family B of region 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:16,982 DEBUG [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B 2023-04-08 02:56:16,982 DEBUG [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B 2023-04-08 02:56:16,982 INFO [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] regionserver.CompactingMemStore(121): Store=B, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 02:56:16,982 INFO [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 337c881227bdfa873bb568857a6d7e4b columnFamilyName B 2023-04-08 02:56:16,983 INFO [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] regionserver.HStore(310): Store=337c881227bdfa873bb568857a6d7e4b/B, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:56:16,983 INFO [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] regionserver.HStore(381): Created cacheConfig: cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false, cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false, prefetchOnOpen=false, for column family C of region 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:16,983 DEBUG [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C 2023-04-08 02:56:16,983 DEBUG [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] util.CommonFSUtils(522): Set storagePolicy=HOT for path=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C 2023-04-08 02:56:16,983 INFO [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] regionserver.CompactingMemStore(121): Store=C, in-memory flush size threshold=38.40 KB, immutable segments index type=ARRAY_MAP, compactor=EAGER, pipelineThreshold=2, compactionCellMax=10 2023-04-08 02:56:16,984 INFO [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] compactions.CompactionConfiguration(173): size [minCompactSize:128 MB, maxCompactSize:8.00 EB, offPeakMaxCompactSize:8.00 EB); files [minFilesToCompact:3, maxFilesToCompact:10); ratio 1.200000; off-peak ratio 5.000000; throttle point 2684354560; major period 604800000, major jitter 0.500000, min locality to compact 0.000000; tiered compaction: max_age 9223372036854775807, incoming window min 6, compaction policy for tiered window org.apache.hadoop.hbase.regionserver.compactions.ExploringCompactionPolicy, single output for minor true, compaction window factory org.apache.hadoop.hbase.regionserver.compactions.ExponentialCompactionWindowFactory, region 337c881227bdfa873bb568857a6d7e4b columnFamilyName C 2023-04-08 02:56:16,984 INFO [StoreOpener-337c881227bdfa873bb568857a6d7e4b-1] regionserver.HStore(310): Store=337c881227bdfa873bb568857a6d7e4b/C, memstore type=CompactingMemStore, storagePolicy=HOT, verifyBulkLoads=false, parallelPutCountPrintThreshold=50, encoding=NONE, compression=NONE 2023-04-08 02:56:16,984 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1146): Setting FlushNonSloppyStoresFirstPolicy for the region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:16,984 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:16,985 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(5209): Found 0 recovered edits file(s) under hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:16,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.FlushLargeStoresPolicy(65): No hbase.hregion.percolumnfamilyflush.size.lower.bound set in table TestAcidGuarantees descriptor;using region.getMemStoreFlushHeapSize/# of families (16.0 M)) instead. 2023-04-08 02:56:16,987 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1055): writing seq id for 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:16,988 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1072): Opened 337c881227bdfa873bb568857a6d7e4b; next sequenceid=5; ConstantSizeRegionSplitPolicy{desiredMaxFileSize=61441471, jitterRate=-0.08445073664188385}, FlushLargeStoresPolicy{flushSizeLowerBound=16777216} 2023-04-08 02:56:16,988 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(965): Region open journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:16,988 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2335): Post open deploy tasks for TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., pid=61, masterSystemTime=1680922576974 2023-04-08 02:56:16,989 DEBUG [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionServer(2362): Finished post open deploy task for TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:16,990 INFO [RS_OPEN_REGION-regionserver/jenkins-hbase4:0-0] handler.AssignRegionHandler(156): Opened TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:16,990 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=59 updating hbase:meta row=337c881227bdfa873bb568857a6d7e4b, regionState=OPEN, openSeqNum=5, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:16,990 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":5,"row":"TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922576990"},{"qualifier":"server","vlen":31,"tag":[],"timestamp":"1680922576990"},{"qualifier":"serverstartcode","vlen":8,"tag":[],"timestamp":"1680922576990"},{"qualifier":"seqnumDuringOpen","vlen":8,"tag":[],"timestamp":"1680922576990"}]},"ts":"1680922576990"} 2023-04-08 02:56:16,991 INFO [PEWorker-1] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=61, resume processing ppid=59 2023-04-08 02:56:16,992 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=61, ppid=59, state=SUCCESS; OpenRegionProcedure 337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 in 168 msec 2023-04-08 02:56:16,993 INFO [PEWorker-4] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=59, resume processing ppid=58 2023-04-08 02:56:16,993 INFO [PEWorker-4] procedure2.ProcedureExecutor(1410): Finished pid=59, ppid=58, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=337c881227bdfa873bb568857a6d7e4b, REOPEN/MOVE in 486 msec 2023-04-08 02:56:16,994 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=58, resume processing ppid=57 2023-04-08 02:56:16,994 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=58, ppid=57, state=SUCCESS; ReopenTableRegionsProcedure table=TestAcidGuarantees in 490 msec 2023-04-08 02:56:16,996 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=57, state=SUCCESS; ModifyTableProcedure table=TestAcidGuarantees in 508 msec 2023-04-08 02:56:16,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=57 2023-04-08 02:56:16,997 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x1644730f to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:56:17,001 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6f06dd50, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:56:17,002 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x6af74c0e to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:56:17,006 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@5f9e8654, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:56:17,006 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x1d759aab to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:56:17,010 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@4db22237, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:56:17,011 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x39241fe3 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:56:17,015 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@8b57aae, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:56:17,015 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x1fbb5b46 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:56:17,019 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@6799455, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:56:17,019 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x66c25427 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:56:17,023 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@2c53bf96, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:56:17,023 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x4f4eab6d to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:56:17,027 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3b59bdcf, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:56:17,027 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x07d997c6 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:56:17,034 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@3d765193, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:56:17,034 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x7993d99e to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:56:17,038 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@32bf16cd, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:56:17,038 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(139): Connect 0x4fae3ed2 to 127.0.0.1:64826 with session timeout=90000ms, retries 30, retry interval 1000ms, keepAlive=60000ms 2023-04-08 02:56:17,041 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@28164190, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=null 2023-04-08 02:56:17,055 DEBUG [hconnection-0x660341a4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:56:17,056 DEBUG [hconnection-0x33056dc4-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:56:17,056 DEBUG [hconnection-0x65e45b43-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:56:17,057 DEBUG [hconnection-0x1c5fe6f-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:56:17,057 DEBUG [hconnection-0x72955698-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:56:17,057 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46594, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:56:17,059 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46582, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:56:17,059 DEBUG [hconnection-0x4e7945cd-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:56:17,059 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46586, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:56:17,059 DEBUG [hconnection-0x1702f866-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:56:17,069 DEBUG [hconnection-0xbc50a61-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:56:17,061 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:17,071 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46618, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:56:17,071 DEBUG [hconnection-0x4249eaf6-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:56:17,060 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46610, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:56:17,060 DEBUG [hconnection-0x7c36a053-metaLookup-shared--pool-0] ipc.RpcConnection(124): Using SIMPLE authentication for service=ClientService, sasl=false 2023-04-08 02:56:17,071 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46634, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:56:17,076 INFO [RS-EventLoopGroup-3-1] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46664, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:56:17,076 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46644, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:56:17,077 INFO [RS-EventLoopGroup-3-2] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46656, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:56:17,077 INFO [RS-EventLoopGroup-3-3] ipc.ServerRpcConnection(540): Connection from 172.31.14.131:46678, version=2.4.18-SNAPSHOT, sasl=false, ugi=jenkins (auth:SIMPLE), service=ClientService 2023-04-08 02:56:17,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:17,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:17,087 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-08 02:56:17,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:17,087 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:17,087 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:17,087 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:17,087 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:17,087 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:17,087 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:17,087 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:17,087 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:17,088 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:17,088 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:17,088 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,088 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:17,089 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,089 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:17,090 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:17,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:17,090 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,090 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:17,090 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:17,090 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:17,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:17,091 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:17,091 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:17,091 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:17,091 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:17,091 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:17,091 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:17,091 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:17,092 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:17,092 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:17,092 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:17,092 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:17,092 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:17,092 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:17,092 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:17,092 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,092 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:17,092 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:17,098 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,098 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:17,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:17,099 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,099 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:17,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:17,099 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:17,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:17,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:17,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:17,100 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6e06c4b6[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:17,100 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:17,101 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,102 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,102 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:17,102 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,102 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,102 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:17,102 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:17,102 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:17,102 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040886fd1993efc54a0a94b06c5fac71018b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040886fd1993efc54a0a94b06c5fac71018b 2023-04-08 02:56:17,104 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:17,104 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:17,104 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:17,104 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:17,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:17,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:17,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:17,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:17,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,106 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:17,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922577101, max timestamp=1680922577106Num uniques 859; 2023-04-08 02:56:17,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 5 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680922637104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:56:17,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922577101, max timestamp=1680922577106Num uniques 859; 2023-04-08 02:56:17,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922577101, max timestamp=1680922577106Num uniques 859; 2023-04-08 02:56:17,107 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:17,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:17,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46594 deadline: 1680922637104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:17,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:17,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:17,109 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:17,109 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:17,109 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:17,110 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:17,110 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:17,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:56:17,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:56:17,112 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,112 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:17,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:17,112 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:17,112 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=16, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/d7a3cf0c2afc4af1b50007bc0f6ecef3 2023-04-08 02:56:17,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:17,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:17,113 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:17,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:17,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:17,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:17,114 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:17,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:17,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,117 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:17,117 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:17,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:17,118 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,118 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:17,118 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:17,118 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,118 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:17,118 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:17,118 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:17,118 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:17,118 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,118 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,120 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:17,120 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:17,120 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:17,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:17,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:17,124 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:17,124 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680922637122, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:17,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 10 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922637123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,125 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:17,125 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46618 deadline: 1680922637123, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,209 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:17,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 7 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680922637209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:17,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46594 deadline: 1680922637209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,220 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:17,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:17,222 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:17,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:17,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:17,223 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:17,223 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:17,223 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:17,223 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:17,223 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,223 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:17,225 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,225 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:17,225 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:17,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:17,225 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,225 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:17,225 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:17,225 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:17,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680922637224, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,225 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:17,226 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:17,226 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:17,226 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:17,226 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:17,226 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:17,226 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:17,226 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:17,227 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:17,227 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46618 deadline: 1680922637226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,228 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:17,228 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 12 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922637226, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,231 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:17,231 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:17,232 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:17,232 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:17,232 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:17,232 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:17,232 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:17,232 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:17,232 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,232 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:17,232 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:17,233 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,234 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:17,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:17,234 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,234 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:17,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:17,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:17,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:17,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:17,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:17,235 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bb37078[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:17,235 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:17,235 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,237 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,237 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:17,237 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:17,237 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,238 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:17,238 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:17,239 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,239 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:17,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:17,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:17,239 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:17,239 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:17,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:17,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:17,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:17,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:17,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:17,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:17,241 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:17,241 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:17,241 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:17,241 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:17,241 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:17,241 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:17,243 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,243 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,243 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:17,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:17,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:17,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:17,243 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:17,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:17,243 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:17,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:17,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:17,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,244 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:17,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:17,246 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,247 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,247 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:17,247 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:17,247 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,247 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:17,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:17,247 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:17,247 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:17,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:17,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:17,247 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:17,247 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:17,247 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:17,247 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,247 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:17,247 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,247 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:17,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:17,347 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:17,348 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:17,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:17,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:17,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:17,352 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:17,352 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:17,352 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:17,352 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:17,353 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,353 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:17,355 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:17,355 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,355 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:17,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:17,355 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,355 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:17,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:17,356 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:17,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:17,356 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:17,357 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:17,357 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:17,357 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:17,357 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:17,357 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:17,359 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:17,359 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:17,359 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:17,359 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:17,360 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:17,360 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:17,360 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:17,360 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:17,360 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,360 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:17,360 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:17,361 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,361 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:17,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:17,362 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:17,362 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:17,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:17,362 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:17,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,363 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:17,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:17,363 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@21202c1f[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:17,363 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:17,363 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,364 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,364 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,364 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,364 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:17,364 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:17,364 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:17,364 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,364 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:17,366 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,366 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:17,366 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:17,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:17,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:17,366 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:17,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:17,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:17,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:17,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:17,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:17,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:17,368 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:17,368 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:17,368 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:17,368 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:17,368 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:17,368 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:17,371 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,371 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:17,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,371 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:17,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:17,371 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:17,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:17,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:17,371 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:17,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:17,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:17,372 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:17,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:17,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,375 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:17,375 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:17,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:17,375 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:17,375 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:17,375 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:17,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:17,375 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:17,375 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:17,375 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:17,375 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:17,375 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,375 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,375 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,375 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:17,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:17,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:17,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 9 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680922637411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,411 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:17,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46594 deadline: 1680922637411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,428 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:17,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680922637427, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,429 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:17,429 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 14 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922637428, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,431 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:17,431 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46618 deadline: 1680922637429, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,475 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:17,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:17,477 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:17,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:17,478 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:17,478 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:17,479 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:17,479 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:17,479 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:17,479 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,479 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:17,480 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,480 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:17,480 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:17,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:17,481 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,481 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:17,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:17,481 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:17,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:17,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:17,481 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:17,482 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:17,482 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:17,482 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:17,482 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:17,482 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:17,482 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:17,482 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:17,482 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:17,482 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:17,482 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:17,482 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:17,482 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:17,482 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,482 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:17,482 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:17,484 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,484 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:17,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:17,484 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,484 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:17,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:17,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:17,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:17,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:17,485 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:17,486 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@379e3a25[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:17,486 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:17,486 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,490 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,490 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,490 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,490 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:17,491 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:17,491 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,491 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:17,491 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:17,492 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,492 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:17,492 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:17,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:17,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:17,492 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:17,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:17,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:17,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:17,493 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:17,494 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:17,495 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:17,495 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:17,495 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:17,495 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:17,495 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:17,495 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:17,496 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,496 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:17,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:17,496 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,496 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:17,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:17,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:17,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:17,497 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:17,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:17,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:17,498 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:17,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:17,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,502 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,502 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:17,502 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:17,502 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:17,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:17,502 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:17,502 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:17,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:17,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:17,502 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:17,502 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:17,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:17,502 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:17,502 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,502 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:17,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:17,502 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:17,502 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,536 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/d62c465e7adf46a28719aed70915c5a1 2023-04-08 02:56:17,561 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=16 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/45b995b0b41a422fa66bab9f110fc233 2023-04-08 02:56:17,566 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/d7a3cf0c2afc4af1b50007bc0f6ecef3 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/d7a3cf0c2afc4af1b50007bc0f6ecef3 2023-04-08 02:56:17,571 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/d7a3cf0c2afc4af1b50007bc0f6ecef3, entries=150, sequenceid=16, filesize=25.1 K 2023-04-08 02:56:17,572 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/d62c465e7adf46a28719aed70915c5a1 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/d62c465e7adf46a28719aed70915c5a1 2023-04-08 02:56:17,584 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/d62c465e7adf46a28719aed70915c5a1, entries=150, sequenceid=16, filesize=11.6 K 2023-04-08 02:56:17,585 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/45b995b0b41a422fa66bab9f110fc233 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/45b995b0b41a422fa66bab9f110fc233 2023-04-08 02:56:17,590 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/45b995b0b41a422fa66bab9f110fc233, entries=150, sequenceid=16, filesize=11.6 K 2023-04-08 02:56:17,591 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 337c881227bdfa873bb568857a6d7e4b in 504ms, sequenceid=16, compaction requested=false 2023-04-08 02:56:17,591 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:17,602 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:17,603 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:17,605 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:17,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:17,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:17,612 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:17,612 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:17,612 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:17,612 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:17,612 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,612 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:17,614 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:17,614 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,614 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:17,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:17,614 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:17,615 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:17,615 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,615 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:17,615 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:17,616 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:17,616 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:17,616 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:17,616 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:17,616 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:17,616 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:17,616 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:17,616 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:17,617 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:17,617 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:17,617 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:17,617 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=120.76 KB heapSize=345.14 KB 2023-04-08 02:56:17,617 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:17,617 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:17,617 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:17,617 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:17,617 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:17,617 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:17,717 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:17,717 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:17,734 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:17,734 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922637732, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:17,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 16 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680922637736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,737 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:17,737 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680922637736, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,738 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:17,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46594 deadline: 1680922637737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,739 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:17,739 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46618 deadline: 1680922637737, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,838 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:17,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 18 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680922637838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:17,841 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:17,841 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46594 deadline: 1680922637839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,040 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:18,040 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 20 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680922638040, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,043 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:18,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 24 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46594 deadline: 1680922638042, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,045 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408fbf3b3224ab54f9883f2440764ead719 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408fbf3b3224ab54f9883f2440764ead719 2023-04-08 02:56:18,049 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=51, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/5cef7f52deb441e5b3530bd9736e475a 2023-04-08 02:56:18,057 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/448b524cc31445aeab7d537e7afbb156 2023-04-08 02:56:18,078 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=51 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/8fabc2dfdfc24b7a838160f26036567e 2023-04-08 02:56:18,084 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/5cef7f52deb441e5b3530bd9736e475a as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/5cef7f52deb441e5b3530bd9736e475a 2023-04-08 02:56:18,087 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/5cef7f52deb441e5b3530bd9736e475a, entries=150, sequenceid=51, filesize=25.1 K 2023-04-08 02:56:18,088 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/448b524cc31445aeab7d537e7afbb156 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/448b524cc31445aeab7d537e7afbb156 2023-04-08 02:56:18,093 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/448b524cc31445aeab7d537e7afbb156, entries=150, sequenceid=51, filesize=11.6 K 2023-04-08 02:56:18,093 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/8fabc2dfdfc24b7a838160f26036567e as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/8fabc2dfdfc24b7a838160f26036567e 2023-04-08 02:56:18,098 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/8fabc2dfdfc24b7a838160f26036567e, entries=150, sequenceid=51, filesize=11.6 K 2023-04-08 02:56:18,099 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~345.09 KB/353376, currentSize=60.38 KB/61830 for 337c881227bdfa873bb568857a6d7e4b in 482ms, sequenceid=51, compaction requested=false 2023-04-08 02:56:18,099 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:18,099 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:18,099 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:18,099 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:18,099 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,104 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:18,104 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:18,106 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,106 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:18,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:18,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:18,107 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:18,107 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:18,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:18,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:18,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:18,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:18,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:18,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6fe57b14[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:18,110 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:18,110 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:18,111 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:18,111 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:18,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:18,111 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:18,111 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:18,111 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:18,111 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,111 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:18,115 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:18,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:18,115 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:18,115 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:18,115 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:18,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:18,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:18,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:18,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:18,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:18,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:18,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:18,118 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:18,118 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:18,118 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:18,118 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:18,118 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:18,122 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:18,122 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:18,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:18,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:18,122 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:18,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:18,122 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:18,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:18,124 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:18,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:18,125 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:18,126 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:18,127 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:18,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,130 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:18,130 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:18,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:18,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:18,130 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:18,130 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:18,130 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,130 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:18,130 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:18,131 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,131 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:18,131 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:18,131 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:18,131 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:18,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:18,131 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:18,131 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:18,131 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:18,131 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:18,231 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:18,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:18,233 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:18,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:18,235 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:18,235 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:18,235 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:18,235 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:18,235 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:18,235 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:18,235 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:18,236 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:18,237 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,237 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:18,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:18,237 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,237 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:18,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:18,237 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:18,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:18,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:18,238 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:18,238 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:18,238 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:18,238 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:18,238 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:18,238 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:18,238 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:18,238 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:18,238 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:18,238 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:18,239 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=60.38 KB heapSize=180.05 KB 2023-04-08 02:56:18,239 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:18,239 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:18,239 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:18,239 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:18,239 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:18,239 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:18,240 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:18,240 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:18,256 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408cf996f4ab5dd4ca3bf398cb0b29a3a83 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408cf996f4ab5dd4ca3bf398cb0b29a3a83 2023-04-08 02:56:18,259 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:18,259 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 27 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46618 deadline: 1680922638256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680922578254, max timestamp=1680922578258Num uniques 859; 2023-04-08 02:56:18,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-08 02:56:18,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680922578254, max timestamp=1680922578258Num uniques 859; 2023-04-08 02:56:18,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9564, totalHeapSize=21976, min timestamp=1680922578254, max timestamp=1680922578258Num uniques 859; 2023-04-08 02:56:18,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-08 02:56:18,260 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39334, new segment data size=9564, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-08 02:56:18,261 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=63, memsize=20.1 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/3b1a4df88f5d470c8a4f5cbed136b6a1 2023-04-08 02:56:18,282 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:18,282 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680922638279, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,284 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:18,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922638282, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,344 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:18,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 22 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680922638342, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:18,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 26 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46594 deadline: 1680922638344, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,363 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:18,363 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46618 deadline: 1680922638361, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,385 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:18,385 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680922638383, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,387 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:18,387 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922638385, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,567 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:18,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46618 deadline: 1680922638565, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,587 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:18,587 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680922638586, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,591 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:18,591 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922638588, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,674 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/f58e638146f94d9daab7142c722f93b2 2023-04-08 02:56:18,687 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=20.13 KB at sequenceid=63 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/112b5fc443b043db8b34b22313a42ff7 2023-04-08 02:56:18,691 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/3b1a4df88f5d470c8a4f5cbed136b6a1 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/3b1a4df88f5d470c8a4f5cbed136b6a1 2023-04-08 02:56:18,695 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/3b1a4df88f5d470c8a4f5cbed136b6a1, entries=150, sequenceid=63, filesize=25.1 K 2023-04-08 02:56:18,696 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/f58e638146f94d9daab7142c722f93b2 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/f58e638146f94d9daab7142c722f93b2 2023-04-08 02:56:18,699 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/f58e638146f94d9daab7142c722f93b2, entries=150, sequenceid=63, filesize=11.6 K 2023-04-08 02:56:18,700 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/112b5fc443b043db8b34b22313a42ff7 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/112b5fc443b043db8b34b22313a42ff7 2023-04-08 02:56:18,703 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/112b5fc443b043db8b34b22313a42ff7, entries=150, sequenceid=63, filesize=11.6 K 2023-04-08 02:56:18,704 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~60.38 KB/61830, heapSize ~180 KB/184320, currentSize=120.76 KB/123660 for 337c881227bdfa873bb568857a6d7e4b in 466ms, sequenceid=63, compaction requested=true 2023-04-08 02:56:18,704 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:18,704 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:18,704 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:18,707 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,707 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:18,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:18,708 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:18,708 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:18,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:18,709 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5f65e620[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:18,709 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:18,709 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:18,710 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:18,710 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:18,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:18,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:18,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:18,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:18,710 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,710 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:18,711 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:18,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:18,711 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:18,712 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:18,712 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:18,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:18,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:18,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:18,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:18,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:18,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:18,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:18,714 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:18,714 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:18,714 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:18,714 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:18,714 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:18,715 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:18,715 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:18,715 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:18,715 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:18,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:18,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:18,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:18,716 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:18,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:18,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:18,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:18,717 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:18,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:18,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:18,720 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,720 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:18,720 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:18,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:18,720 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:18,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:18,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:18,720 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:18,720 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:18,720 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:18,720 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:18,720 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:18,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:18,720 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:18,720 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:18,720 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,720 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:18,720 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:18,720 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:18,820 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:18,821 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:18,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:18,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:18,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:18,824 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:18,824 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:18,824 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:18,824 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:18,824 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:18,824 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:18,825 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:18,825 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,825 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:18,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:18,826 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,826 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:18,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:18,826 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:18,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:18,826 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:18,827 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:18,827 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:18,827 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:18,827 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:18,827 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:18,827 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:18,827 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:18,827 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:18,827 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:18,827 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:18,828 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=120.76 KB heapSize=340.45 KB 2023-04-08 02:56:18,828 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:18,828 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:18,828 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:18,828 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:18,828 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:18,828 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:18,845 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304085682a570759748a2a20a17da58749b76 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304085682a570759748a2a20a17da58749b76 2023-04-08 02:56:18,850 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:18,850 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:18,853 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=97, memsize=40.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/c70d804639fa4d6a93344ae5668cc7c5 2023-04-08 02:56:18,863 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/53fb381951a0431baf4c122852dc4c6d 2023-04-08 02:56:18,865 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:18,865 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 32 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46594 deadline: 1680922638862, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,867 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:18,867 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 29 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680922638864, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,872 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:18,872 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46618 deadline: 1680922638870, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,889 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:18,890 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680922638888, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,894 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:18,894 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922638893, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,967 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:18,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 34 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46594 deadline: 1680922638966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:18,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:18,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 31 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680922638968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:19,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 36 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46594 deadline: 1680922639170, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,171 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:19,171 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 33 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680922639171, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,201 WARN [HBase-Metrics2-1] impl.MetricsConfig(128): Cannot locate configuration: tried hadoop-metrics2-hbase.properties,hadoop-metrics2.properties 2023-04-08 02:56:19,272 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=40.25 KB at sequenceid=97 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/7a58b6eadcd94426926df15b6ea55587 2023-04-08 02:56:19,276 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/c70d804639fa4d6a93344ae5668cc7c5 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/c70d804639fa4d6a93344ae5668cc7c5 2023-04-08 02:56:19,283 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/c70d804639fa4d6a93344ae5668cc7c5, entries=150, sequenceid=97, filesize=25.1 K 2023-04-08 02:56:19,284 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/53fb381951a0431baf4c122852dc4c6d as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/53fb381951a0431baf4c122852dc4c6d 2023-04-08 02:56:19,289 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/53fb381951a0431baf4c122852dc4c6d, entries=150, sequenceid=97, filesize=11.6 K 2023-04-08 02:56:19,290 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/7a58b6eadcd94426926df15b6ea55587 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/7a58b6eadcd94426926df15b6ea55587 2023-04-08 02:56:19,298 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/7a58b6eadcd94426926df15b6ea55587, entries=150, sequenceid=97, filesize=11.6 K 2023-04-08 02:56:19,299 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~120.76 KB/123660, heapSize ~340.41 KB/348576, currentSize=60.38 KB/61830 for 337c881227bdfa873bb568857a6d7e4b in 471ms, sequenceid=97, compaction requested=true 2023-04-08 02:56:19,299 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:19,300 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:19,300 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:19,300 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:19,300 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,300 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:19,300 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:19,301 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,301 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,301 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:19,302 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,302 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:19,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:19,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,304 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:19,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:19,304 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7f057907[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:19,304 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:19,304 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,306 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,306 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,306 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:19,306 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:19,306 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:19,306 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,306 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:19,308 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:19,308 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:19,308 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:19,308 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:19,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:19,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:19,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:19,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,310 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:19,312 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:19,312 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:19,312 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:19,312 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:19,312 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:19,312 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:19,317 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,317 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:19,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,317 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:19,317 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,318 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:19,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:19,319 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,321 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,322 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:19,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,323 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,323 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,328 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:19,328 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,328 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:19,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:19,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:19,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:19,329 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:19,329 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:19,329 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:19,329 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:19,329 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:19,329 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:19,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:19,329 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,329 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:19,329 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,329 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,329 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,375 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:19,375 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-08 02:56:19,375 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:19,375 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:19,375 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:19,376 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:19,376 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:19,376 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:19,392 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408abb64fb8ceb04cb891c9b41cf2bf4b63 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408abb64fb8ceb04cb891c9b41cf2bf4b63 2023-04-08 02:56:19,412 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:19,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922639408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,414 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:19,414 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46618 deadline: 1680922639411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,417 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:19,417 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 45 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680922639413, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,429 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:19,430 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:19,432 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:19,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:19,447 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:19,447 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:19,447 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:19,447 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:19,447 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:19,448 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,448 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:19,448 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=110, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/4bf7b447754f447ea4f4f97999956081 2023-04-08 02:56:19,449 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:19,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:19,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:19,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:19,450 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,450 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,450 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,450 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:19,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:19,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:19,451 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:19,452 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:19,452 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:19,452 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:19,452 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:19,452 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:19,452 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:19,452 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:19,452 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:19,452 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:19,452 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:19,453 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:19,453 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:19,453 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:19,453 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,453 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:19,453 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:19,454 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:19,454 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:19,454 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:19,455 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,455 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:19,455 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,456 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:19,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:19,457 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@390c2bd0[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:19,457 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:19,457 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,459 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,460 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:19,460 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:19,460 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:19,460 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,460 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:19,462 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,462 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:19,462 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:19,462 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:19,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:19,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:19,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:19,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:19,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:19,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,465 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:19,466 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:19,466 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:19,466 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:19,466 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:19,466 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:19,466 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:19,468 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,468 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,468 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:19,468 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:19,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:19,469 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:19,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:19,470 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,471 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:19,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:19,473 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:19,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 38 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46594 deadline: 1680922639472, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,474 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,474 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:19,474 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:19,474 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,474 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,474 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,474 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,474 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:19,474 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:19,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:19,474 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:19,475 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:19,475 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:19,475 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:19,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:19,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:19,475 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:19,475 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:19,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:19,476 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:19,476 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 35 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680922639475, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,478 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/31753b646c1c4d77ac26de8ffe8d614b 2023-04-08 02:56:19,516 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:19,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 44 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922639514, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,517 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:19,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46618 deadline: 1680922639515, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,518 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=110 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/6ef17dd2bd4c44d089c44fcc2492035f 2023-04-08 02:56:19,521 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:19,522 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680922639518, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,524 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/4bf7b447754f447ea4f4f97999956081 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/4bf7b447754f447ea4f4f97999956081 2023-04-08 02:56:19,529 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/4bf7b447754f447ea4f4f97999956081, entries=200, sequenceid=110, filesize=31.9 K 2023-04-08 02:56:19,530 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/31753b646c1c4d77ac26de8ffe8d614b as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/31753b646c1c4d77ac26de8ffe8d614b 2023-04-08 02:56:19,533 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/31753b646c1c4d77ac26de8ffe8d614b, entries=150, sequenceid=110, filesize=11.6 K 2023-04-08 02:56:19,534 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/6ef17dd2bd4c44d089c44fcc2492035f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/6ef17dd2bd4c44d089c44fcc2492035f 2023-04-08 02:56:19,540 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/6ef17dd2bd4c44d089c44fcc2492035f, entries=150, sequenceid=110, filesize=11.6 K 2023-04-08 02:56:19,541 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=107.34 KB/109920 for 337c881227bdfa873bb568857a6d7e4b in 166ms, sequenceid=110, compaction requested=true 2023-04-08 02:56:19,541 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:19,541 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:19,541 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:19,541 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 02:56:19,541 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 02:56:19,541 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:19,542 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 135670 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 02:56:19,542 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 59430 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 02:56:19,542 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 337c881227bdfa873bb568857a6d7e4b/A is initiating minor compaction (all files) 2023-04-08 02:56:19,543 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 337c881227bdfa873bb568857a6d7e4b/B is initiating minor compaction (all files) 2023-04-08 02:56:19,543 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 337c881227bdfa873bb568857a6d7e4b/A in TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:19,543 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 337c881227bdfa873bb568857a6d7e4b/B in TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:19,543 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/d7a3cf0c2afc4af1b50007bc0f6ecef3, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/5cef7f52deb441e5b3530bd9736e475a, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/3b1a4df88f5d470c8a4f5cbed136b6a1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/c70d804639fa4d6a93344ae5668cc7c5, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/4bf7b447754f447ea4f4f97999956081] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp, totalSize=132.5 K 2023-04-08 02:56:19,543 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/d62c465e7adf46a28719aed70915c5a1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/448b524cc31445aeab7d537e7afbb156, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/f58e638146f94d9daab7142c722f93b2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/53fb381951a0431baf4c122852dc4c6d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/31753b646c1c4d77ac26de8ffe8d614b] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp, totalSize=58.0 K 2023-04-08 02:56:19,543 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting d7a3cf0c2afc4af1b50007bc0f6ecef3, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680922577071 2023-04-08 02:56:19,543 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting d62c465e7adf46a28719aed70915c5a1, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680922577071 2023-04-08 02:56:19,543 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 5cef7f52deb441e5b3530bd9736e475a, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680922577115 2023-04-08 02:56:19,543 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 448b524cc31445aeab7d537e7afbb156, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680922577115 2023-04-08 02:56:19,543 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 3b1a4df88f5d470c8a4f5cbed136b6a1, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1680922577727 2023-04-08 02:56:19,544 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting f58e638146f94d9daab7142c722f93b2, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1680922577727 2023-04-08 02:56:19,544 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting c70d804639fa4d6a93344ae5668cc7c5, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1680922578268 2023-04-08 02:56:19,544 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 53fb381951a0431baf4c122852dc4c6d, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1680922578268 2023-04-08 02:56:19,544 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 4bf7b447754f447ea4f4f97999956081, keycount=200, bloomtype=ROW, size=31.9 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1680922578857 2023-04-08 02:56:19,544 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 31753b646c1c4d77ac26de8ffe8d614b, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1680922578857 2023-04-08 02:56:19,554 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 337c881227bdfa873bb568857a6d7e4b#B#compaction#591 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:19,575 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:19,575 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:19,578 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:19,582 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 337c881227bdfa873bb568857a6d7e4b#A#compaction#590 average throughput is 0.66 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:19,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:19,583 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:19,583 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:19,583 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:19,583 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:19,583 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:19,583 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,584 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:19,586 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,586 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:19,586 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:19,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:19,586 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,587 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:19,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:19,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:19,588 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:19,596 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:19,596 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:19,596 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:19,596 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:19,599 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:19,599 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:19,599 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:19,599 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:19,599 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-08 02:56:19,599 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:19,600 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:19,600 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:19,600 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:19,600 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:19,600 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:19,649 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408315b92ec38b3428e92b2b44bbf87c752 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408315b92ec38b3428e92b2b44bbf87c752 2023-04-08 02:56:19,663 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=129, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/7dcb76f2ac7b47e3986eb18323a007d5 2023-04-08 02:56:19,677 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/c537fa9f84c94493bcf8ffc2d0c6ff61 2023-04-08 02:56:19,690 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=129 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/7c558cbd014543658c38040d133718d6 2023-04-08 02:56:19,693 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/7dcb76f2ac7b47e3986eb18323a007d5 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/7dcb76f2ac7b47e3986eb18323a007d5 2023-04-08 02:56:19,697 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/7dcb76f2ac7b47e3986eb18323a007d5, entries=150, sequenceid=129, filesize=25.1 K 2023-04-08 02:56:19,698 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/c537fa9f84c94493bcf8ffc2d0c6ff61 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/c537fa9f84c94493bcf8ffc2d0c6ff61 2023-04-08 02:56:19,702 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/c537fa9f84c94493bcf8ffc2d0c6ff61, entries=150, sequenceid=129, filesize=11.6 K 2023-04-08 02:56:19,702 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/7c558cbd014543658c38040d133718d6 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/7c558cbd014543658c38040d133718d6 2023-04-08 02:56:19,707 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/7c558cbd014543658c38040d133718d6, entries=150, sequenceid=129, filesize=11.6 K 2023-04-08 02:56:19,708 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=0 B/0 for 337c881227bdfa873bb568857a6d7e4b in 109ms, sequenceid=129, compaction requested=true 2023-04-08 02:56:19,708 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:19,708 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:19,708 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:19,708 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:19,708 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,708 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:19,709 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:19,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,710 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:19,710 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:19,711 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,711 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:19,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:19,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:19,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:19,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:19,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7df5ca3e[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:19,712 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:19,713 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,714 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,714 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,714 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:19,714 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:19,714 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:19,714 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,714 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:19,715 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:19,715 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:19,715 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:19,715 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:19,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:19,725 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:19,725 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:19,726 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:19,726 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:19,726 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:19,726 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:19,729 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,729 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,729 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:19,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:19,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:19,729 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,729 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:19,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:19,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:19,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:19,731 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:19,732 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,735 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:19,735 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,735 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:19,735 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:19,735 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:19,735 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:19,735 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,735 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,735 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,735 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:19,735 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:19,735 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:19,735 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:19,735 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:19,736 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:19,736 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:19,736 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:19,736 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:19,763 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:19,763 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-08 02:56:19,764 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:19,764 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:19,764 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:19,764 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:19,764 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:19,765 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:19,776 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408cdf63845527c454d9cc6a97d6d3acf37 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408cdf63845527c454d9cc6a97d6d3acf37 2023-04-08 02:56:19,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680922579808, max timestamp=1680922579818Num uniques 859; 2023-04-08 02:56:19,819 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-08 02:56:19,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680922579808, max timestamp=1680922579818Num uniques 859; 2023-04-08 02:56:19,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-08 02:56:19,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=259, cellSize=11855, totalHeapSize=27176, min timestamp=1680922579808, max timestamp=1680922579818Num uniques 859; 2023-04-08 02:56:19,820 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=11855, suffix heap size=117064,new segment heap size=27176 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=259 2023-04-08 02:56:19,836 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:19,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:19,838 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:19,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:19,839 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:19,839 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:19,839 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:19,839 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:19,839 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:19,839 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,840 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:19,841 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:19,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:19,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:19,841 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,841 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,841 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:19,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:19,842 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:19,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:19,842 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:19,842 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:19,842 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:19,842 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:19,842 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46618 deadline: 1680922639838, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,842 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:19,843 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:19,843 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:19,843 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:19,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 60 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922639839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,843 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:19,843 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:19,843 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:19,843 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 61 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680922639839, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,843 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:19,843 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:19,844 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:19,844 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:19,844 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:19,844 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:19,844 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,844 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:19,844 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:19,845 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,845 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:19,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:19,846 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,846 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:19,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:19,847 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dd16746[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:19,847 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:19,847 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,849 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,849 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,849 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:19,849 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:19,849 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:19,849 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,849 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:19,851 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,851 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:19,851 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:19,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:19,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:19,851 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:19,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:19,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:19,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:19,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,853 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:19,853 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:19,853 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:19,853 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:19,853 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:19,853 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:19,853 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:19,855 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,855 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:19,855 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,855 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:19,855 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:19,856 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:19,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:19,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:19,856 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,856 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,857 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:19,857 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,858 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,859 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,860 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:19,865 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,865 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:19,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:19,865 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:19,865 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:19,865 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:19,866 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:19,865 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:19,866 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:19,866 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,866 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,866 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,866 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,868 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:19,868 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:19,868 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:19,868 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:19,869 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:19,869 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:19,945 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:19,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46618 deadline: 1680922639944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,946 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:19,946 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 62 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922639944, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,948 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:19,948 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 63 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680922639945, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,966 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:19,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:19,968 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:19,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:19,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:19,969 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:19,969 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:19,969 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:19,969 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:19,969 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,970 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:19,971 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:19,971 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:19,971 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,971 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:19,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:19,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:19,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:19,972 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:19,973 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:19,973 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:19,973 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:19,973 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:19,973 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:19,973 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:19,973 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:19,973 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:19,973 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:19,973 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:19,973 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:19,973 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:19,973 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:19,973 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,973 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:19,974 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:19,975 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,975 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:19,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:19,976 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,976 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:19,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:19,978 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:19,979 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5948faba[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:19,979 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:19,979 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,979 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:19,980 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 40 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46594 deadline: 1680922639976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,980 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:19,981 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 37 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680922639979, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,981 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,981 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,981 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:19,981 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:19,981 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:19,981 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,981 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:19,985 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,985 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:19,985 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:19,985 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:19,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:19,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:19,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:19,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:19,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,987 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:19,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,988 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,989 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:19,989 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:19,989 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:19,989 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:19,989 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:19,989 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:19,989 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:19,991 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,991 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:19,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,991 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/0fe03742967447ff918a37718e65e1de as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/0fe03742967447ff918a37718e65e1de 2023-04-08 02:56:19,991 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:19,991 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:19,991 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,992 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,992 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,993 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:19,994 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,994 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:19,995 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:19,996 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 337c881227bdfa873bb568857a6d7e4b/B of 337c881227bdfa873bb568857a6d7e4b into 0fe03742967447ff918a37718e65e1de(size=11.8 K), total size for store is 23.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:19,996 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:19,996 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., storeName=337c881227bdfa873bb568857a6d7e4b/B, priority=11, startTime=1680922579541; duration=0sec 2023-04-08 02:56:19,996 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:19,996 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 6 store files, 0 compacting, 6 eligible, 16 blocking 2023-04-08 02:56:19,996 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,997 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,997 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:19,997 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:19,998 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:19,998 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:19,998 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:19,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:19,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:19,998 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,998 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:19,998 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 6 files of size 71316 starting at candidate #0 after considering 10 permutations with 10 in ratio 2023-04-08 02:56:19,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:19,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:19,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:19,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:19,998 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:19,998 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:19,999 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:19,999 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:19,999 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 337c881227bdfa873bb568857a6d7e4b/C is initiating minor compaction (all files) 2023-04-08 02:56:19,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:19,999 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 337c881227bdfa873bb568857a6d7e4b/C in TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:19,999 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/45b995b0b41a422fa66bab9f110fc233, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/8fabc2dfdfc24b7a838160f26036567e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/112b5fc443b043db8b34b22313a42ff7, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/7a58b6eadcd94426926df15b6ea55587, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/6ef17dd2bd4c44d089c44fcc2492035f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/7c558cbd014543658c38040d133718d6] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp, totalSize=69.6 K 2023-04-08 02:56:19,999 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 45b995b0b41a422fa66bab9f110fc233, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=16, earliestPutTs=1680922577071 2023-04-08 02:56:20,000 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 8fabc2dfdfc24b7a838160f26036567e, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=51, earliestPutTs=1680922577115 2023-04-08 02:56:20,000 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 112b5fc443b043db8b34b22313a42ff7, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=63, earliestPutTs=1680922577727 2023-04-08 02:56:20,000 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 7a58b6eadcd94426926df15b6ea55587, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=97, earliestPutTs=1680922578268 2023-04-08 02:56:20,001 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 6ef17dd2bd4c44d089c44fcc2492035f, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1680922578857 2023-04-08 02:56:20,001 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 7c558cbd014543658c38040d133718d6, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1680922579399 2023-04-08 02:56:20,011 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 337c881227bdfa873bb568857a6d7e4b#C#compaction#596 average throughput is 2.18 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:20,099 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:20,099 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:20,101 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:20,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:20,102 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:20,102 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:20,102 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:20,102 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:20,102 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:20,103 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,103 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:20,104 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:20,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:20,104 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:20,104 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,104 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:20,104 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:20,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:20,105 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:20,105 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:20,105 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:20,105 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:20,105 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:20,105 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:20,105 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:20,105 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:20,105 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:20,105 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:20,105 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:20,106 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:20,106 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:20,106 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:20,106 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,106 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:20,106 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:20,108 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,108 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:20,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:20,108 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,108 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:20,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:20,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:20,110 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@14218e8[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:20,111 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:20,111 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,112 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,112 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:20,112 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:20,112 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:20,112 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,112 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:20,113 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,114 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:20,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:20,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:20,114 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:20,114 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:20,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:20,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:20,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:20,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:20,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:20,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:20,118 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:20,118 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:20,118 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:20,118 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:20,118 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:20,118 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:20,120 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,120 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,120 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:20,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:20,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:20,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:20,121 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:20,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:20,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:20,121 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,121 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:20,122 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,123 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:20,123 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:20,124 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,127 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,127 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:20,127 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:20,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:20,127 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:20,127 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:20,127 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:20,127 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,127 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:20,128 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,128 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,128 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,127 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:20,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:20,128 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:20,128 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:20,128 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:20,128 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:20,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:20,152 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 65 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680922640151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,152 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:20,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46618 deadline: 1680922640151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,153 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:20,153 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 64 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922640151, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,182 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=140, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/7869dde691b845dda59e07aca00dfb07 2023-04-08 02:56:20,212 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/919d2585676d45e9b911d664023d5fdd 2023-04-08 02:56:20,228 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:20,229 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:20,230 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:20,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:20,231 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:20,231 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:20,231 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:20,231 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:20,231 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:20,232 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,232 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:20,233 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,233 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:20,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:20,233 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,233 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:20,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:20,234 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:20,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:20,234 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:20,234 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:20,234 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:20,234 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:20,234 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:20,234 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:20,234 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:20,234 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:20,234 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:20,234 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:20,234 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:20,235 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:20,235 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:20,235 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:20,235 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,235 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:20,235 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:20,236 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,236 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:20,237 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,237 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:20,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:20,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:20,238 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52b596c4[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:20,238 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:20,238 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,239 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,239 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:20,240 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:20,240 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,240 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:20,240 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:20,242 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,242 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:20,242 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:20,242 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:20,242 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:20,243 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:20,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:20,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:20,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,245 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:20,245 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:20,245 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:20,245 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:20,245 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:20,245 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:20,245 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:20,247 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:20,247 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,247 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:20,247 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:20,247 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,247 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:20,248 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:20,248 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:20,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,249 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,249 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:20,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:20,250 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,256 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,256 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:20,256 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:20,256 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:20,256 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:20,256 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:20,256 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,256 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:20,256 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,256 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:20,256 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:20,256 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:20,256 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:20,256 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:20,256 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,257 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:20,257 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:20,357 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:20,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:20,366 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:20,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:20,367 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:20,367 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:20,367 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:20,367 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:20,367 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:20,367 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,367 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:20,369 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,369 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:20,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:20,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:20,370 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,370 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:20,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:20,370 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:20,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:20,370 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:20,370 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:20,371 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:20,371 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:20,371 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:20,371 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:20,372 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:20,372 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:20,372 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:20,372 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:20,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:20,374 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,374 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:20,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:20,375 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,375 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:20,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:20,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:20,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:20,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:20,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:20,376 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@54583083[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:20,376 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:20,376 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,377 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,377 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,377 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:20,377 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:20,377 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:20,378 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,378 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:20,379 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:20,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:20,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:20,379 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:20,380 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:20,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:20,380 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:20,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:20,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,380 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:20,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:20,381 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,382 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:20,382 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:20,382 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:20,382 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:20,382 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:20,382 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:20,382 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:20,385 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,385 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:20,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,385 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:20,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:20,385 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:20,385 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:20,386 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:20,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:20,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:20,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:20,392 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,392 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:20,393 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:20,394 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,395 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,395 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:20,395 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:20,395 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,395 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:20,395 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:20,395 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:20,395 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:20,395 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:20,396 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,396 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,396 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,396 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:20,396 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:20,395 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:20,396 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:20,395 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:20,397 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:20,397 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:20,420 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/4c3952ef594543229ece4804f89cce9e as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/4c3952ef594543229ece4804f89cce9e 2023-04-08 02:56:20,425 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/1ee939c85e3b4b4f8d4503a57d519a82 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/1ee939c85e3b4b4f8d4503a57d519a82 2023-04-08 02:56:20,425 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 6 (all) file(s) in 337c881227bdfa873bb568857a6d7e4b/C of 337c881227bdfa873bb568857a6d7e4b into 4c3952ef594543229ece4804f89cce9e(size=11.8 K), total size for store is 11.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:20,425 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:20,425 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., storeName=337c881227bdfa873bb568857a6d7e4b/C, priority=10, startTime=1680922579541; duration=0sec 2023-04-08 02:56:20,425 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:20,430 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 337c881227bdfa873bb568857a6d7e4b/A of 337c881227bdfa873bb568857a6d7e4b into 1ee939c85e3b4b4f8d4503a57d519a82(size=25.3 K), total size for store is 50.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:20,430 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:20,430 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., storeName=337c881227bdfa873bb568857a6d7e4b/A, priority=11, startTime=1680922579541; duration=0sec 2023-04-08 02:56:20,430 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:20,453 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:20,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 67 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680922640453, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,456 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:20,456 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 66 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922640455, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,457 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:20,457 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 68 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46618 deadline: 1680922640456, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,497 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:20,497 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:20,499 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:20,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:20,500 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:20,500 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:20,500 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:20,500 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:20,500 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:20,501 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,501 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:20,502 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:20,502 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,502 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:20,502 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,502 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:20,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:20,502 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:20,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:20,503 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:20,503 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:20,503 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:20,503 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:20,503 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:20,503 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:20,503 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:20,504 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:20,504 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:20,504 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:20,504 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:20,504 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:20,504 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:20,504 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:20,504 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,504 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:20,504 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:20,505 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,505 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,505 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:20,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:20,506 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,506 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:20,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:20,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:20,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:20,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:20,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:20,508 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4a5b9d98[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:20,508 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:20,508 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,510 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,510 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,510 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:20,510 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,510 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:20,510 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:20,510 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:20,512 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,512 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:20,512 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:20,512 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:20,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:20,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:20,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:20,514 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:20,514 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:20,514 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:20,514 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:20,514 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:20,514 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:20,516 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,516 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:20,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:20,516 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,516 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:20,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:20,517 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,517 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:20,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:20,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:20,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:20,517 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:20,517 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,518 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:20,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:20,519 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,520 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,520 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:20,520 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:20,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:20,520 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:20,520 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:20,520 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,520 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,520 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:20,520 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:20,520 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:20,520 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:20,520 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:20,520 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:20,520 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:20,520 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:20,520 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:20,521 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:20,620 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:20,621 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:20,622 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=140 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/bb267508ba034d348146a4bc32cb6091 2023-04-08 02:56:20,623 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:20,626 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/7869dde691b845dda59e07aca00dfb07 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/7869dde691b845dda59e07aca00dfb07 2023-04-08 02:56:20,630 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/7869dde691b845dda59e07aca00dfb07, entries=350, sequenceid=140, filesize=52.6 K 2023-04-08 02:56:20,631 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/919d2585676d45e9b911d664023d5fdd as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/919d2585676d45e9b911d664023d5fdd 2023-04-08 02:56:20,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:20,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:20,632 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:20,632 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:20,632 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:20,632 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:20,632 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,633 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:20,635 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:20,635 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:20,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:20,635 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,635 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/919d2585676d45e9b911d664023d5fdd, entries=150, sequenceid=140, filesize=11.8 K 2023-04-08 02:56:20,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:20,635 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,635 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:20,636 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:20,636 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/bb267508ba034d348146a4bc32cb6091 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/bb267508ba034d348146a4bc32cb6091 2023-04-08 02:56:20,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:20,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:20,636 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:20,656 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:20,656 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:20,656 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:20,656 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:20,656 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:20,658 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/bb267508ba034d348146a4bc32cb6091, entries=150, sequenceid=140, filesize=11.8 K 2023-04-08 02:56:20,659 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 337c881227bdfa873bb568857a6d7e4b in 896ms, sequenceid=140, compaction requested=true 2023-04-08 02:56:20,659 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:20,659 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:20,659 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:56:20,659 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:20,659 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:20,660 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:20,660 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:20,660 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 105524 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:56:20,660 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 337c881227bdfa873bb568857a6d7e4b/A is initiating minor compaction (all files) 2023-04-08 02:56:20,660 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 337c881227bdfa873bb568857a6d7e4b/A in TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:20,661 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/1ee939c85e3b4b4f8d4503a57d519a82, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/7dcb76f2ac7b47e3986eb18323a007d5, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/7869dde691b845dda59e07aca00dfb07] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp, totalSize=103.1 K 2023-04-08 02:56:20,661 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 1ee939c85e3b4b4f8d4503a57d519a82, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1680922578857 2023-04-08 02:56:20,661 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 7dcb76f2ac7b47e3986eb18323a007d5, keycount=150, bloomtype=ROW, size=25.1 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1680922579399 2023-04-08 02:56:20,662 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 7869dde691b845dda59e07aca00dfb07, keycount=350, bloomtype=ROW, size=52.6 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680922579737 2023-04-08 02:56:20,659 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:20,660 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:56:20,664 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:20,664 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=127.47 KB heapSize=355.69 KB 2023-04-08 02:56:20,664 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:20,665 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:20,665 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:20,665 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:20,665 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:20,665 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:20,666 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 35980 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:56:20,666 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 337c881227bdfa873bb568857a6d7e4b/B is initiating minor compaction (all files) 2023-04-08 02:56:20,666 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 337c881227bdfa873bb568857a6d7e4b/B in TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:20,666 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/0fe03742967447ff918a37718e65e1de, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/c537fa9f84c94493bcf8ffc2d0c6ff61, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/919d2585676d45e9b911d664023d5fdd] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp, totalSize=35.1 K 2023-04-08 02:56:20,666 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 0fe03742967447ff918a37718e65e1de, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=110, earliestPutTs=1680922578857 2023-04-08 02:56:20,667 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting c537fa9f84c94493bcf8ffc2d0c6ff61, keycount=150, bloomtype=ROW, size=11.6 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1680922579399 2023-04-08 02:56:20,667 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 919d2585676d45e9b911d664023d5fdd, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680922579757 2023-04-08 02:56:20,676 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 337c881227bdfa873bb568857a6d7e4b#B#compaction#600 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:20,685 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 337c881227bdfa873bb568857a6d7e4b#A#compaction#599 average throughput is 1.10 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:20,707 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/e66558ba4c1b4bb2b4bfbd92fced47de as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/e66558ba4c1b4bb2b4bfbd92fced47de 2023-04-08 02:56:20,712 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 337c881227bdfa873bb568857a6d7e4b/B of 337c881227bdfa873bb568857a6d7e4b into e66558ba4c1b4bb2b4bfbd92fced47de(size=12.0 K), total size for store is 12.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:20,712 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:20,712 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., storeName=337c881227bdfa873bb568857a6d7e4b/B, priority=13, startTime=1680922580660; duration=0sec 2023-04-08 02:56:20,712 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:20,712 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 2 store files, 0 compacting, 2 eligible, 16 blocking 2023-04-08 02:56:20,713 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 0 files of size 0 starting at candidate #-1 after considering 0 permutations with 0 in ratio 2023-04-08 02:56:20,713 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(232): Not compacting files because we only have 0 files ready for compaction. Need 3 to initiate. 2023-04-08 02:56:20,713 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit(415): Not compacting TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. because compaction request was cancelled 2023-04-08 02:56:20,717 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408aed15bb58d984994b642562305aef6af to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408aed15bb58d984994b642562305aef6af 2023-04-08 02:56:20,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:20,959 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:20,968 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:20,968 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46618 deadline: 1680922640966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,969 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:20,969 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:20,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680922640966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,969 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 71 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922640966, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,985 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:20,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 39 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680922640984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:20,986 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:20,986 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 42 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46594 deadline: 1680922640984, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:21,071 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:21,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46618 deadline: 1680922641069, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:21,071 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:21,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922641070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:21,071 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:21,071 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680922641070, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:21,121 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=177, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/6d22543d880e4655bb7b91dc4c4eeb08 2023-04-08 02:56:21,274 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:21,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922641273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:21,275 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:21,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46618 deadline: 1680922641273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:21,275 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:21,275 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680922641273, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:21,526 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/839e455c04cf4259affa82b7690c6c28 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/839e455c04cf4259affa82b7690c6c28 2023-04-08 02:56:21,529 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/2287672a7c114fa595e115aa7a4b35ce 2023-04-08 02:56:21,531 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 337c881227bdfa873bb568857a6d7e4b/A of 337c881227bdfa873bb568857a6d7e4b into 839e455c04cf4259affa82b7690c6c28(size=25.6 K), total size for store is 25.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:21,531 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:21,531 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., storeName=337c881227bdfa873bb568857a6d7e4b/A, priority=13, startTime=1680922580659; duration=0sec 2023-04-08 02:56:21,532 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:21,577 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:21,577 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922641576, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:21,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:21,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46618 deadline: 1680922641577, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:21,579 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:21,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 78 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680922641578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:21,939 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=177 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/a5e8a0d098f944d8815ddeaff0f6973e 2023-04-08 02:56:21,943 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/6d22543d880e4655bb7b91dc4c4eeb08 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/6d22543d880e4655bb7b91dc4c4eeb08 2023-04-08 02:56:21,947 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/6d22543d880e4655bb7b91dc4c4eeb08, entries=150, sequenceid=177, filesize=25.3 K 2023-04-08 02:56:21,947 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/2287672a7c114fa595e115aa7a4b35ce as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/2287672a7c114fa595e115aa7a4b35ce 2023-04-08 02:56:21,951 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/2287672a7c114fa595e115aa7a4b35ce, entries=150, sequenceid=177, filesize=11.8 K 2023-04-08 02:56:21,951 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/a5e8a0d098f944d8815ddeaff0f6973e as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/a5e8a0d098f944d8815ddeaff0f6973e 2023-04-08 02:56:21,954 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/a5e8a0d098f944d8815ddeaff0f6973e, entries=150, sequenceid=177, filesize=11.8 K 2023-04-08 02:56:21,955 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~355.64 KB/364176, currentSize=53.67 KB/54960 for 337c881227bdfa873bb568857a6d7e4b in 1291ms, sequenceid=177, compaction requested=true 2023-04-08 02:56:21,955 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:21,955 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:21,955 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:21,955 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:21,955 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:21,955 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:21,955 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:21,956 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:21,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:21,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:21,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:21,956 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:21,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:21,957 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:21,957 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:21,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:21,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:21,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:21,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:21,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:21,958 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5b4f4963[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:21,958 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:21,958 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:21,959 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:21,959 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:21,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:21,959 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:21,959 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:21,960 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:21,960 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:21,960 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:21,961 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:21,961 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:21,961 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:21,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:21,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:21,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:21,961 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:21,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:21,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:21,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:21,963 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:21,963 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:21,964 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:21,964 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:21,964 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:21,964 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:21,966 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:21,966 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:21,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:21,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:21,966 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:21,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:21,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:21,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:21,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:21,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:21,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:21,967 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:21,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:21,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:21,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:21,968 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:21,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:21,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:21,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:21,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:21,971 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:21,971 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:21,971 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:21,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:21,971 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:21,971 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:21,972 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:21,971 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:21,971 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:21,972 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:21,972 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:21,972 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:21,972 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:21,972 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:21,972 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:21,972 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:21,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:21,972 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:21,972 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:22,072 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:22,072 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:22,074 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:22,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:22,075 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:22,075 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:22,075 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:22,075 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:22,075 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:22,076 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:22,076 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:22,077 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:22,077 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,077 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:22,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:22,077 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,077 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:22,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:22,078 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:22,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:22,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:22,078 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:22,078 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:22,078 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:22,078 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:22,078 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:22,078 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:22,079 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:22,079 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:22,079 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:22,079 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:22,079 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-08 02:56:22,079 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:22,079 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:22,079 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:22,079 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:22,079 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:22,079 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:22,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:22,080 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:22,091 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408c910b35944d7473082d05bb188ea87a2 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408c910b35944d7473082d05bb188ea87a2 2023-04-08 02:56:22,107 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:22,107 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 86 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922642104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,107 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:22,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922582092, max timestamp=1680922582106Num uniques 859; 2023-04-08 02:56:22,108 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680922642104, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922582092, max timestamp=1680922582106Num uniques 859; 2023-04-08 02:56:22,107 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922582092, max timestamp=1680922582106Num uniques 859; 2023-04-08 02:56:22,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:56:22,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:56:22,108 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:56:22,159 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:22,159 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46618 deadline: 1680922642156, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:22,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 88 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922642208, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,210 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:22,210 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680922642209, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,261 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:22,262 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46618 deadline: 1680922642260, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:22,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 90 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922642411, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,413 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:22,413 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 89 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680922642412, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,464 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:22,464 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46618 deadline: 1680922642463, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,501 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=190, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/1681a8bb820d4a83ad933f5d26ab2fa3 2023-04-08 02:56:22,715 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:22,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 92 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922642714, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,716 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:22,716 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 91 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680922642715, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,768 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:22,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46618 deadline: 1680922642767, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,912 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/8114c671901b47c5a55bb29066daf063 2023-04-08 02:56:22,927 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=190 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/25ec35eb369c47e0a2eaddef8f5a346d 2023-04-08 02:56:22,931 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/1681a8bb820d4a83ad933f5d26ab2fa3 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/1681a8bb820d4a83ad933f5d26ab2fa3 2023-04-08 02:56:22,935 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/1681a8bb820d4a83ad933f5d26ab2fa3, entries=150, sequenceid=190, filesize=25.3 K 2023-04-08 02:56:22,936 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/8114c671901b47c5a55bb29066daf063 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/8114c671901b47c5a55bb29066daf063 2023-04-08 02:56:22,949 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/8114c671901b47c5a55bb29066daf063, entries=150, sequenceid=190, filesize=11.8 K 2023-04-08 02:56:22,950 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/25ec35eb369c47e0a2eaddef8f5a346d as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/25ec35eb369c47e0a2eaddef8f5a346d 2023-04-08 02:56:22,953 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/25ec35eb369c47e0a2eaddef8f5a346d, entries=150, sequenceid=190, filesize=11.8 K 2023-04-08 02:56:22,954 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 337c881227bdfa873bb568857a6d7e4b in 875ms, sequenceid=190, compaction requested=true 2023-04-08 02:56:22,954 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:22,954 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:22,954 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:22,954 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:22,954 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,954 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:22,954 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:22,965 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,965 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:22,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:22,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:22,965 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:22,965 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:22,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:22,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:22,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2532e778[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:22,967 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:22,967 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:22,968 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:22,968 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:22,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:22,968 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:22,968 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:22,968 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:22,968 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,968 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:22,969 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:22,970 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,970 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:22,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:22,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:22,970 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:22,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:22,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:22,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:22,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:22,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:22,971 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:22,972 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:22,972 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:22,972 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:22,972 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:22,972 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:22,972 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:22,973 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:22,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:22,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:22,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:22,974 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:22,974 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:22,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:22,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:22,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:22,974 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,974 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:22,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:22,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:22,975 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:22,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:22,976 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:22,978 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:22,978 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:22,978 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,978 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:22,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:22,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:22,978 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:22,979 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:22,979 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:22,979 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:22,979 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:22,979 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:22,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:22,979 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:22,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:22,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:22,979 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:22,979 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:22,979 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:23,002 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:23,003 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-08 02:56:23,003 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:23,003 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:23,003 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:23,003 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:23,003 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:23,003 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:23,017 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304081e4d045903c9465189e15c0fbc77e10d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304081e4d045903c9465189e15c0fbc77e10d 2023-04-08 02:56:23,026 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:23,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 47 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46594 deadline: 1680922643022, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,027 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:23,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 46 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680922643024, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,079 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:23,080 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:23,082 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:23,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:23,084 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:23,084 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:23,084 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:23,084 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:23,084 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:23,085 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,085 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,086 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:23,086 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,086 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:23,086 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:23,086 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,086 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:23,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,087 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:23,087 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,087 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:23,088 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:23,088 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:23,088 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:23,088 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:23,089 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:23,089 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:23,089 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:23,089 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:23,089 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:23,090 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:23,090 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:23,090 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:23,090 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:23,090 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,090 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:23,090 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:23,091 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,091 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:23,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:23,091 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,091 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:23,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:23,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:23,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:23,095 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:23,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30fb0775[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:23,096 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:23,096 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,098 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,098 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:23,098 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:23,099 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:23,099 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,099 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:23,101 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:23,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:23,102 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:23,102 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:23,102 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:23,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:23,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,103 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:23,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:23,105 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:23,105 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:23,105 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:23,105 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:23,105 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:23,105 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:23,107 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,107 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,107 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,107 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:23,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:23,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:23,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:23,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:23,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:23,109 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:23,110 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,111 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,111 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:23,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,112 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,114 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,114 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:23,114 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:23,114 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:23,114 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,114 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,114 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,114 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,114 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,114 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:23,114 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:23,114 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:23,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:23,114 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:23,115 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:23,115 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:23,115 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:23,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:23,129 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:23,130 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 49 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46594 deadline: 1680922643129, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,132 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:23,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 48 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680922643130, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,215 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:23,215 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:23,217 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:23,218 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:23,219 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 93 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680922643217, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:23,220 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:23,220 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:23,220 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:23,220 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:23,220 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:23,220 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,221 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,222 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,222 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:23,222 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:23,222 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,222 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:23,222 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,222 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,222 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:23,223 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 94 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922643220, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:23,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,223 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:23,223 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:23,223 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:23,223 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:23,223 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:23,223 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:23,227 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:23,227 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:23,227 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:23,227 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:23,227 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:23,227 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:23,227 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:23,227 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:23,227 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,227 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:23,227 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:23,230 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,230 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:23,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:23,230 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,230 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:23,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:23,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:23,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,231 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:23,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:23,232 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@26dafbaf[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:23,232 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:23,232 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,233 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,233 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:23,233 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:23,233 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:23,233 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,233 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:23,234 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,234 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:23,234 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:23,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:23,234 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:23,237 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:23,237 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:23,237 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:23,237 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:23,237 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:23,237 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:23,238 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,238 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,238 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,238 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:23,238 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:23,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:23,239 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:23,239 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:23,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:23,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,239 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:23,240 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,241 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:23,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,241 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,243 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,243 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,243 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:23,243 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:23,243 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:23,243 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:23,243 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,243 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:23,244 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:23,244 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,244 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:23,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:23,244 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,244 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:23,244 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:23,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:23,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:23,273 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:23,273 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46618 deadline: 1680922643270, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:23,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 51 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46594 deadline: 1680922643332, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,338 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:23,338 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 50 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680922643335, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,344 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:23,344 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:23,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:23,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:23,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:23,347 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:23,347 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:23,347 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:23,347 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:23,347 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,347 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,348 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:23,349 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,349 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:23,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:23,349 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,349 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:23,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,349 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,349 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:23,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,350 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:23,350 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:23,350 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:23,350 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:23,350 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:23,350 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:23,356 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:23,356 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:23,356 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:23,356 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:23,356 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:23,356 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:23,356 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:23,356 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:23,356 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,356 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:23,356 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:23,359 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:23,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:23,359 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:23,359 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,359 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:23,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:23,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:23,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:23,360 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@77faa46[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:23,360 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:23,360 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,370 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,370 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,370 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:23,370 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,370 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:23,370 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:23,370 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:23,371 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:23,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:23,371 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:23,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:23,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:23,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:23,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:23,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:23,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:23,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:23,374 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:23,374 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:23,374 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:23,374 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:23,374 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:23,375 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,375 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,375 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:23,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:23,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:23,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,375 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:23,376 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:23,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:23,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,376 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:23,377 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,377 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:23,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,378 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,379 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,379 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:23,379 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:23,379 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:23,379 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:23,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:23,379 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:23,379 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:23,379 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:23,379 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:23,379 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,379 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:23,379 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:23,379 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,379 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:23,379 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,380 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,474 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=227, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/5faacd544c4141098f59ab143c0bf3e0 2023-04-08 02:56:23,479 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:23,480 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:23,482 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:23,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:23,488 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:23,488 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:23,488 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:23,488 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:23,488 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:23,488 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/b907ccacf17547bdae4390659a7cf622 2023-04-08 02:56:23,488 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,489 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,491 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:23,491 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:23,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:23,491 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,491 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:23,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,491 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:23,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,492 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:23,493 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:23,493 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:23,493 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:23,493 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:23,493 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:23,493 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:23,493 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:23,493 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:23,493 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:23,493 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:23,493 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:23,493 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:23,493 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:23,493 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,494 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:23,494 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:23,495 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,495 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:23,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:23,495 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,495 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:23,495 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:23,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:23,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,496 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:23,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:23,497 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@354494f6[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:23,497 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:23,497 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,498 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,498 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,499 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,499 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:23,499 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:23,499 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:23,499 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,499 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:23,500 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:23,500 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:23,500 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,500 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:23,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:23,500 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:23,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:23,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:23,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,501 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:23,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:23,503 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:23,503 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:23,503 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:23,503 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:23,503 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:23,503 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:23,505 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,505 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,506 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:23,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:23,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:23,507 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:23,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,509 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:23,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,512 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,512 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,512 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,512 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:23,512 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:23,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:23,512 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:23,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:23,512 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:23,512 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:23,512 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,512 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,512 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,512 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:23,512 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:23,512 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:23,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:23,512 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:23,512 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,613 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:23,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:23,615 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:23,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:23,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:23,616 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:23,616 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:23,616 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:23,616 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:23,616 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,616 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,618 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,618 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:23,618 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:23,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:23,618 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,618 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:23,618 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,619 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:23,619 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,619 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:23,619 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:23,619 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:23,619 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:23,619 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:23,619 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:23,628 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:23,628 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:23,628 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:23,628 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:23,628 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:23,628 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:23,628 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:23,628 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:23,628 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,628 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:23,628 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:23,630 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:23,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:23,630 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:23,631 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,631 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,631 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:23,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:23,632 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5deced84[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:23,632 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:23,632 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,634 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,634 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,634 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:23,634 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:23,635 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:23,635 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,635 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:23,636 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:23,636 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:23,636 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:23,636 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:23,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:23,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:23,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:23,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:23,638 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:23,639 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:23,639 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:23,639 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:23,639 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 53 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46594 deadline: 1680922643638, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,639 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:23,639 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:23,639 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:23,639 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:23,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:23,641 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 52 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680922643640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,641 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:23,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,642 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:23,642 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,642 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:23,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:23,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,644 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:23,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,646 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:23,646 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:23,646 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:23,646 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:23,646 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,646 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:23,646 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,646 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:23,647 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:23,646 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,647 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,647 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,647 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,647 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,646 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:23,647 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:23,647 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:23,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:23,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:23,747 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:23,748 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:23,749 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:23,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:23,750 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:23,750 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:23,750 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:23,750 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:23,750 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:23,751 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,751 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,753 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:23,753 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,753 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:23,753 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:23,753 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,753 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:23,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,754 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:23,754 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,754 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:23,754 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:23,754 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:23,754 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:23,754 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:23,755 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:23,755 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:23,755 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:23,755 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:23,755 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:23,755 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:23,755 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:23,755 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:23,755 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:23,755 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,755 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:23,755 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:23,757 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,757 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:23,757 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:23,758 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,758 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:23,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:23,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:23,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,758 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:23,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:23,759 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2c1a7215[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:23,759 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:23,759 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,760 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,760 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:23,761 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,761 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,761 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:23,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:23,761 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:23,762 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:23,762 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,762 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:23,762 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:23,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,763 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:23,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,764 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:23,764 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:23,764 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:23,764 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:23,764 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:23,764 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:23,764 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:23,769 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,769 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,769 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:23,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,770 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:23,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:23,770 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:23,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:23,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,771 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:23,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,773 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,773 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:23,773 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,773 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:23,773 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:23,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:23,773 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:23,773 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,773 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:23,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:23,774 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:23,774 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:23,773 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:23,773 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,774 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,774 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,774 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,774 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:23,774 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:23,874 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:23,874 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:23,875 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:23,876 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:23,877 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:23,877 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:23,877 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:23,877 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:23,877 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:23,877 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,877 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,878 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,878 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:23,878 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:23,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:23,879 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,879 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:23,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,879 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:23,879 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,879 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:23,880 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:23,880 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:23,880 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:23,880 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:23,880 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:23,880 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:23,880 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:23,880 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:23,880 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:23,880 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:23,880 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:23,880 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:23,880 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:23,880 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,880 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:23,880 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:23,883 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,883 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:23,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:23,883 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,883 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:23,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:23,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:23,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:23,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:23,884 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@564dc04a[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:23,884 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:23,884 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,885 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,885 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,885 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:23,885 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:23,885 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:23,886 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,886 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:23,887 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,887 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:23,887 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:23,887 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:23,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:23,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:23,889 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:23,889 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:23,889 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:23,889 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:23,889 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:23,889 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:23,890 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,890 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,890 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,890 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:23,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:23,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:23,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:23,891 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:23,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,892 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:23,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:23,893 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,895 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:23,895 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,895 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:23,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:23,895 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:23,895 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:23,895 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:23,895 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:23,895 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:23,895 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:23,895 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:23,895 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:23,895 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:23,895 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,896 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:23,896 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:23,898 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=227 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/3b0affee2c7b4ff2808d951de9fef160 2023-04-08 02:56:23,901 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/5faacd544c4141098f59ab143c0bf3e0 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/5faacd544c4141098f59ab143c0bf3e0 2023-04-08 02:56:23,905 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/5faacd544c4141098f59ab143c0bf3e0, entries=200, sequenceid=227, filesize=32.1 K 2023-04-08 02:56:23,905 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/b907ccacf17547bdae4390659a7cf622 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/b907ccacf17547bdae4390659a7cf622 2023-04-08 02:56:23,909 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/b907ccacf17547bdae4390659a7cf622, entries=150, sequenceid=227, filesize=11.8 K 2023-04-08 02:56:23,910 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/3b0affee2c7b4ff2808d951de9fef160 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/3b0affee2c7b4ff2808d951de9fef160 2023-04-08 02:56:23,912 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/3b0affee2c7b4ff2808d951de9fef160, entries=150, sequenceid=227, filesize=11.8 K 2023-04-08 02:56:23,913 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 337c881227bdfa873bb568857a6d7e4b in 910ms, sequenceid=227, compaction requested=true 2023-04-08 02:56:23,913 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:23,913 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:23,913 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:23,913 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:56:23,913 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 02:56:23,913 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=2), splitQueue=0 2023-04-08 02:56:23,915 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 110848 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:56:23,915 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 337c881227bdfa873bb568857a6d7e4b/A is initiating minor compaction (all files) 2023-04-08 02:56:23,915 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 337c881227bdfa873bb568857a6d7e4b/A in TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:23,915 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/839e455c04cf4259affa82b7690c6c28, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/6d22543d880e4655bb7b91dc4c4eeb08, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/1681a8bb820d4a83ad933f5d26ab2fa3, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/5faacd544c4141098f59ab143c0bf3e0] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp, totalSize=108.3 K 2023-04-08 02:56:23,915 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 60236 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 02:56:23,915 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 839e455c04cf4259affa82b7690c6c28, keycount=150, bloomtype=ROW, size=25.6 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680922579757 2023-04-08 02:56:23,915 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 337c881227bdfa873bb568857a6d7e4b/C is initiating minor compaction (all files) 2023-04-08 02:56:23,915 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 337c881227bdfa873bb568857a6d7e4b/C in TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:23,915 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/4c3952ef594543229ece4804f89cce9e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/bb267508ba034d348146a4bc32cb6091, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/a5e8a0d098f944d8815ddeaff0f6973e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/25ec35eb369c47e0a2eaddef8f5a346d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/3b0affee2c7b4ff2808d951de9fef160] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp, totalSize=58.8 K 2023-04-08 02:56:23,916 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 6d22543d880e4655bb7b91dc4c4eeb08, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1680922579826 2023-04-08 02:56:23,916 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 4c3952ef594543229ece4804f89cce9e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=129, earliestPutTs=1680922579399 2023-04-08 02:56:23,916 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 1681a8bb820d4a83ad933f5d26ab2fa3, keycount=150, bloomtype=ROW, size=25.3 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1680922580961 2023-04-08 02:56:23,917 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting bb267508ba034d348146a4bc32cb6091, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680922579757 2023-04-08 02:56:23,917 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 5faacd544c4141098f59ab143c0bf3e0, keycount=200, bloomtype=ROW, size=32.1 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680922582149 2023-04-08 02:56:23,917 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting a5e8a0d098f944d8815ddeaff0f6973e, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1680922579826 2023-04-08 02:56:23,917 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 25ec35eb369c47e0a2eaddef8f5a346d, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1680922580961 2023-04-08 02:56:23,918 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 3b0affee2c7b4ff2808d951de9fef160, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680922582152 2023-04-08 02:56:23,927 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 337c881227bdfa873bb568857a6d7e4b#C#compaction#611 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:23,930 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 337c881227bdfa873bb568857a6d7e4b#A#compaction#610 average throughput is 3.29 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:23,996 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:23,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:23,998 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:23,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:23,999 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:23,999 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:23,999 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:23,999 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:23,999 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:24,000 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:24,000 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:24,001 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:24,001 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,001 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:24,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:24,001 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,001 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:24,001 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:24,001 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:24,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:24,002 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:24,002 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:24,002 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:24,002 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:24,002 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:24,002 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:24,002 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:24,002 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:24,002 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:24,002 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:24,003 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:24,003 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-08 02:56:24,003 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:24,003 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:24,003 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:24,004 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:24,004 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:24,004 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:24,147 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:24,147 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:24,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922584172, max timestamp=1680922584178Num uniques 859; 2023-04-08 02:56:24,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:56:24,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922584172, max timestamp=1680922584178Num uniques 859; 2023-04-08 02:56:24,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:56:24,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922584172, max timestamp=1680922584178Num uniques 859; 2023-04-08 02:56:24,179 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:56:24,222 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:24,222 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 70 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680922644219, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,225 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:24,225 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 73 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46594 deadline: 1680922644222, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,232 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:24,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 95 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680922644229, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,233 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:24,233 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 96 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922644231, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,283 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:24,284 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46618 deadline: 1680922644281, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,326 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:24,327 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 75 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46594 deadline: 1680922644326, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,328 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:24,329 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 72 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680922644328, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,339 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/558b7d7471224163a60dd4e6e30a34eb as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/558b7d7471224163a60dd4e6e30a34eb 2023-04-08 02:56:24,343 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 337c881227bdfa873bb568857a6d7e4b/C of 337c881227bdfa873bb568857a6d7e4b into 558b7d7471224163a60dd4e6e30a34eb(size=12.1 K), total size for store is 12.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:24,343 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:24,343 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., storeName=337c881227bdfa873bb568857a6d7e4b/C, priority=11, startTime=1680922583913; duration=0sec 2023-04-08 02:56:24,343 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:24,343 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:56:24,346 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 48418 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:56:24,346 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 337c881227bdfa873bb568857a6d7e4b/B is initiating minor compaction (all files) 2023-04-08 02:56:24,347 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 337c881227bdfa873bb568857a6d7e4b/B in TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:24,347 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/e66558ba4c1b4bb2b4bfbd92fced47de, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/2287672a7c114fa595e115aa7a4b35ce, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/8114c671901b47c5a55bb29066daf063, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/b907ccacf17547bdae4390659a7cf622] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp, totalSize=47.3 K 2023-04-08 02:56:24,347 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting e66558ba4c1b4bb2b4bfbd92fced47de, keycount=150, bloomtype=ROW, size=12.0 K, encoding=NONE, compression=NONE, seqNum=140, earliestPutTs=1680922579757 2023-04-08 02:56:24,348 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 2287672a7c114fa595e115aa7a4b35ce, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=177, earliestPutTs=1680922579826 2023-04-08 02:56:24,348 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 8114c671901b47c5a55bb29066daf063, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=190, earliestPutTs=1680922580961 2023-04-08 02:56:24,348 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting b907ccacf17547bdae4390659a7cf622, keycount=150, bloomtype=ROW, size=11.8 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680922582152 2023-04-08 02:56:24,354 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 337c881227bdfa873bb568857a6d7e4b#B#compaction#613 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:24,357 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/7ae31d6bc02141edbbd2ad92db56764c as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/7ae31d6bc02141edbbd2ad92db56764c 2023-04-08 02:56:24,360 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 337c881227bdfa873bb568857a6d7e4b/A of 337c881227bdfa873bb568857a6d7e4b into 7ae31d6bc02141edbbd2ad92db56764c(size=25.7 K), total size for store is 25.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:24,360 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:24,360 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., storeName=337c881227bdfa873bb568857a6d7e4b/A, priority=12, startTime=1680922583913; duration=0sec 2023-04-08 02:56:24,360 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:24,421 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408c3a618c6a790409a9cb20ecbe2bd9fb6 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408c3a618c6a790409a9cb20ecbe2bd9fb6 2023-04-08 02:56:24,430 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=237, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/12e56920ca58499c84a41c069b51f1f4 2023-04-08 02:56:24,438 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/16b5beffcd694d7ea1679b6d773f305d 2023-04-08 02:56:24,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:24,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 77 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46594 deadline: 1680922644528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,533 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:24,533 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 74 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680922644531, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,767 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/dbb6ea05f4f34015aa28c353d6e34347 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/dbb6ea05f4f34015aa28c353d6e34347 2023-04-08 02:56:24,771 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 337c881227bdfa873bb568857a6d7e4b/B of 337c881227bdfa873bb568857a6d7e4b into dbb6ea05f4f34015aa28c353d6e34347(size=12.2 K), total size for store is 12.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:24,771 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:24,771 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., storeName=337c881227bdfa873bb568857a6d7e4b/B, priority=12, startTime=1680922583913; duration=0sec 2023-04-08 02:56:24,771 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:24,834 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:24,834 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 79 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46594 deadline: 1680922644833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:24,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 76 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680922644835, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,854 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=237 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/14cb6dab080b41cba5f417d7faf51f00 2023-04-08 02:56:24,858 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/12e56920ca58499c84a41c069b51f1f4 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/12e56920ca58499c84a41c069b51f1f4 2023-04-08 02:56:24,862 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/12e56920ca58499c84a41c069b51f1f4, entries=100, sequenceid=237, filesize=18.5 K 2023-04-08 02:56:24,863 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/16b5beffcd694d7ea1679b6d773f305d as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/16b5beffcd694d7ea1679b6d773f305d 2023-04-08 02:56:24,867 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/16b5beffcd694d7ea1679b6d773f305d, entries=100, sequenceid=237, filesize=9.4 K 2023-04-08 02:56:24,868 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/14cb6dab080b41cba5f417d7faf51f00 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/14cb6dab080b41cba5f417d7faf51f00 2023-04-08 02:56:24,872 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/14cb6dab080b41cba5f417d7faf51f00, entries=100, sequenceid=237, filesize=9.4 K 2023-04-08 02:56:24,872 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 337c881227bdfa873bb568857a6d7e4b in 869ms, sequenceid=237, compaction requested=false 2023-04-08 02:56:24,873 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:24,873 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:24,873 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:24,873 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:24,873 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,873 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:24,873 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:24,875 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,875 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:24,875 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:24,876 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:24,876 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:24,876 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:24,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:24,877 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,878 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:24,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:24,878 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@473f0d51[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:24,878 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:24,878 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:24,880 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:24,880 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:24,880 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:24,880 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:24,880 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:24,880 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:24,880 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,881 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:24,882 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:24,882 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,882 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:24,882 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:24,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:24,883 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:24,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:24,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:24,883 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:24,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:24,884 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:24,885 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,886 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:24,886 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:24,886 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:24,886 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:24,886 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:24,886 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:24,886 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:24,887 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:24,887 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:24,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:24,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:24,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:24,887 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:24,887 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:24,887 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:24,888 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:24,888 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:24,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:24,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:24,888 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:24,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:24,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:24,889 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,889 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:24,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:24,890 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,891 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,891 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:24,891 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:24,891 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:24,891 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:24,891 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:24,891 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:24,891 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,891 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:24,891 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:24,891 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:24,892 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:24,891 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:24,892 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:24,892 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:24,892 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:24,892 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:24,892 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:24,892 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:24,992 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:24,993 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:24,994 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:24,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:24,996 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:24,996 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:24,996 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:24,996 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:24,996 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:24,996 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:24,996 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:24,998 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:24,998 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,998 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:24,998 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:24,998 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:24,998 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:24,999 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:24,999 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:24,999 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:25,000 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:25,000 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:25,000 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:25,000 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:25,000 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:25,000 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:25,000 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:25,000 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:25,000 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:25,001 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=134.18 KB heapSize=384.98 KB 2023-04-08 02:56:25,001 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:25,001 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:25,001 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:25,001 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:25,001 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:25,001 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:25,340 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:25,340 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:25,350 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:25,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 85 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46594 deadline: 1680922645349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,351 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:25,351 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 81 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680922645349, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,431 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408e04b5d52fa57477080fcaa461a69642e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408e04b5d52fa57477080fcaa461a69642e 2023-04-08 02:56:25,435 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=277, memsize=44.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/6816d59a1f704876b767774761f45c26 2023-04-08 02:56:25,442 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/688d702440754c58a86acd68cec74032 2023-04-08 02:56:25,449 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=44.73 KB at sequenceid=277 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/51524096644e44bbb5d3095f857ed383 2023-04-08 02:56:25,453 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/6816d59a1f704876b767774761f45c26 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/6816d59a1f704876b767774761f45c26 2023-04-08 02:56:25,454 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:25,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 87 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46594 deadline: 1680922645452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,455 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:25,455 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 83 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680922645452, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,456 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/6816d59a1f704876b767774761f45c26, entries=150, sequenceid=277, filesize=25.4 K 2023-04-08 02:56:25,456 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/688d702440754c58a86acd68cec74032 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/688d702440754c58a86acd68cec74032 2023-04-08 02:56:25,461 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/688d702440754c58a86acd68cec74032, entries=150, sequenceid=277, filesize=11.9 K 2023-04-08 02:56:25,462 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/51524096644e44bbb5d3095f857ed383 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/51524096644e44bbb5d3095f857ed383 2023-04-08 02:56:25,466 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/51524096644e44bbb5d3095f857ed383, entries=150, sequenceid=277, filesize=11.9 K 2023-04-08 02:56:25,467 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~134.18 KB/137400, heapSize ~384.94 KB/394176, currentSize=46.96 KB/48090 for 337c881227bdfa873bb568857a6d7e4b in 466ms, sequenceid=277, compaction requested=true 2023-04-08 02:56:25,467 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:25,467 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:25,467 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:25,467 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:25,467 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,468 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:25,468 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:25,469 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:25,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:25,470 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:25,470 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,470 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:25,470 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:25,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:25,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,471 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:25,471 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:25,472 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7d0558ff[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:25,472 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:25,472 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,473 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,473 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,473 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:25,473 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:25,473 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:25,473 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,473 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:25,475 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:25,475 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,475 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:25,475 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:25,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:25,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:25,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:25,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:25,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,478 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:25,478 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:25,478 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:25,478 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:25,478 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:25,478 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:25,478 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:25,481 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,481 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:25,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,481 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:25,481 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:25,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:25,482 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,482 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:25,483 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,484 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,484 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:25,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:25,485 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,491 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:25,491 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:25,491 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:25,491 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:25,491 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:25,491 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:25,491 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:25,491 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:25,491 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,491 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,491 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,491 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,491 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:25,491 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:25,491 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:25,492 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:25,492 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:25,592 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:25,592 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:25,594 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:25,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:25,595 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:25,595 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:25,595 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:25,595 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:25,595 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:25,596 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,596 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:25,597 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:25,597 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,597 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:25,597 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,597 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:25,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:25,598 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:25,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:25,598 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:25,599 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:25,599 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:25,599 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:25,599 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:25,599 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:25,599 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:25,599 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:25,599 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:25,599 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:25,599 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-08 02:56:25,599 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:25,600 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:25,600 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:25,600 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:25,600 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:25,600 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:25,612 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304084001c7fa447b4cd6a03d9f77029f02a8 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304084001c7fa447b4cd6a03d9f77029f02a8 2023-04-08 02:56:25,617 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=287, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/f6a99ba0e8f446888fed4a9cd8b08c88 2023-04-08 02:56:25,625 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/bade4b55b34a4875a26af36be0514c73 2023-04-08 02:56:25,634 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=287 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/350c2a97fe214687ba7668954a74e352 2023-04-08 02:56:25,637 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/f6a99ba0e8f446888fed4a9cd8b08c88 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/f6a99ba0e8f446888fed4a9cd8b08c88 2023-04-08 02:56:25,641 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/f6a99ba0e8f446888fed4a9cd8b08c88, entries=150, sequenceid=287, filesize=25.4 K 2023-04-08 02:56:25,641 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/bade4b55b34a4875a26af36be0514c73 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/bade4b55b34a4875a26af36be0514c73 2023-04-08 02:56:25,644 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/bade4b55b34a4875a26af36be0514c73, entries=150, sequenceid=287, filesize=11.9 K 2023-04-08 02:56:25,644 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/350c2a97fe214687ba7668954a74e352 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/350c2a97fe214687ba7668954a74e352 2023-04-08 02:56:25,647 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/350c2a97fe214687ba7668954a74e352, entries=150, sequenceid=287, filesize=11.9 K 2023-04-08 02:56:25,648 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=0 B/0 for 337c881227bdfa873bb568857a6d7e4b in 49ms, sequenceid=287, compaction requested=true 2023-04-08 02:56:25,648 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:25,648 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:25,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:25,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:25,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:25,648 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:25,650 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,650 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:25,650 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:25,651 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,651 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:25,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:25,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:25,652 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2700790[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:25,652 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:25,652 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,654 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,654 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,654 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,654 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:25,654 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,655 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:25,654 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:25,655 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:25,656 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:25,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:25,656 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:25,656 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:25,656 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:25,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:25,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:25,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:25,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:25,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:25,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,658 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:25,658 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:25,659 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:25,659 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:25,659 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:25,659 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:25,659 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:25,660 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,660 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:25,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:25,660 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:25,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:25,661 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:25,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:25,661 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:25,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:25,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:25,662 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:25,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:25,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,664 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:25,664 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,664 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:25,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:25,664 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:25,664 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:25,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:25,664 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:25,665 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:25,665 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:25,664 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:25,665 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:25,665 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,665 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,665 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,665 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,664 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:25,665 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:25,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:25,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:25,685 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-08 02:56:25,685 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:25,685 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:25,685 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:25,685 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:25,685 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:25,685 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:25,694 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408204724480c914ff98baac8bcbfb691d3 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408204724480c914ff98baac8bcbfb691d3 2023-04-08 02:56:25,699 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=298, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/360a984c7d6e4db19c326cf4f62a1c9c 2023-04-08 02:56:25,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680922585707, max timestamp=1680922585714Num uniques 859; 2023-04-08 02:56:25,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680922585707, max timestamp=1680922585714Num uniques 859; 2023-04-08 02:56:25,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-08 02:56:25,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-08 02:56:25,715 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=209, cellSize=9565, totalHeapSize=21976, min timestamp=1680922585707, max timestamp=1680922585714Num uniques 859; 2023-04-08 02:56:25,716 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=9565, suffix heap size=117064,new segment heap size=21976 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=209 2023-04-08 02:56:25,743 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:25,743 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46594 deadline: 1680922645741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,744 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:25,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680922645741, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,765 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:25,765 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:25,767 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:25,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:25,768 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:25,768 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:25,768 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:25,768 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:25,768 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:25,768 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,768 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:25,771 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,771 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:25,771 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:25,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:25,771 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,771 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:25,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:25,771 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:25,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:25,772 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:25,772 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:25,772 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:25,772 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:25,772 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:25,772 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:25,773 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:25,773 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:25,773 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:25,773 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:25,773 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:25,773 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:25,773 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:25,773 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:25,773 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,773 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:25,773 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:25,775 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:25,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:25,775 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:25,775 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,775 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:25,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:25,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:25,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:25,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:25,777 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@3a07c804[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:25,777 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:25,777 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,778 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,778 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,778 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,778 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:25,778 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:25,778 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:25,778 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,778 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:25,779 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:25,779 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:25,779 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,780 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:25,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:25,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:25,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:25,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:25,780 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:25,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:25,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:25,781 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:25,782 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:25,782 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:25,782 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:25,782 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:25,782 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:25,782 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:25,784 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:25,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:25,785 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,785 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:25,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:25,785 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,785 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,786 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:25,786 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:25,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:25,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:25,793 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:25,794 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:25,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:25,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,796 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:25,796 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,797 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:25,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:25,797 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:25,797 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:25,797 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:25,796 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:25,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:25,797 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:25,796 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:25,797 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:25,797 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:25,797 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,797 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,797 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,797 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:25,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:25,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:25,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46594 deadline: 1680922645844, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,846 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:25,846 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 107 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680922645845, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,897 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:25,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:25,899 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:25,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:25,900 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:25,900 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:25,900 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:25,900 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:25,901 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:25,901 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,901 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:25,902 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:25,902 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,902 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:25,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:25,902 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,902 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:25,902 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:25,903 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:25,903 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:25,903 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:25,903 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:25,903 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:25,903 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:25,903 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:25,903 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:25,903 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:25,904 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:25,904 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:25,904 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:25,904 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:25,904 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:25,904 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:25,904 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:25,904 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,904 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:25,904 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:25,905 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,905 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:25,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:25,906 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,906 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:25,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:25,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:25,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:25,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:25,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:25,908 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@56d0f90[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:25,908 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:25,908 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,910 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,910 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,911 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:25,911 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:25,911 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:25,911 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,911 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:25,912 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,912 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:25,912 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:25,912 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:25,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:25,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:25,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:25,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:25,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:25,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:25,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:25,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:25,915 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:25,915 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:25,915 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:25,915 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:25,915 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:25,915 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:25,922 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,922 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,922 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:25,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:25,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:25,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:25,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:25,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:25,922 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:25,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:25,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:25,924 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,924 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:25,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:25,925 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,939 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,939 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:25,939 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:25,939 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:25,939 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:25,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:25,939 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:25,939 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:25,940 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:25,939 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,940 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:25,940 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:25,940 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:25,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:25,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:25,940 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:25,940 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:25,940 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:25,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:26,040 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:26,041 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:26,042 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:26,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:26,043 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:26,043 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:26,043 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:26,043 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:26,043 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:26,043 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,043 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:26,046 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:26,046 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,046 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:26,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:26,046 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,046 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:26,046 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:26,047 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:26,047 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:26,047 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:26,047 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:26,047 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:26,047 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:26,047 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:26,047 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:26,048 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:26,048 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:26,048 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:26,048 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:26,048 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:26,048 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:26,048 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:26,048 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:26,048 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,048 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:26,048 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:26,049 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:26,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46594 deadline: 1680922646047, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,049 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:26,049 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 109 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680922646048, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,049 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,049 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,049 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:26,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:26,050 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,050 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:26,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:26,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:26,050 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:26,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:26,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:26,051 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@52f87d62[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:26,051 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:26,051 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,052 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,052 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,052 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:26,053 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:26,053 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:26,053 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,053 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:26,054 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,054 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:26,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:26,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:26,054 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:26,054 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:26,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:26,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:26,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:26,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:26,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:26,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:26,057 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:26,057 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:26,057 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:26,057 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:26,057 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:26,057 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:26,059 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,059 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:26,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:26,059 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:26,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:26,060 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:26,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:26,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:26,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,060 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:26,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:26,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:26,061 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:26,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,065 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:26,065 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:26,065 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,065 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:26,065 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:26,065 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,065 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,065 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,065 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:26,065 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:26,065 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:26,065 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:26,065 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:26,066 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:26,066 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:26,066 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:26,066 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:26,107 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/dd6f598366374a00a2c4f51fae2d4c9c 2023-04-08 02:56:26,114 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=298 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/42a545fd2fc94998a0da12cea2797844 2023-04-08 02:56:26,117 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/360a984c7d6e4db19c326cf4f62a1c9c as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/360a984c7d6e4db19c326cf4f62a1c9c 2023-04-08 02:56:26,120 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/360a984c7d6e4db19c326cf4f62a1c9c, entries=200, sequenceid=298, filesize=32.3 K 2023-04-08 02:56:26,121 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/dd6f598366374a00a2c4f51fae2d4c9c as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/dd6f598366374a00a2c4f51fae2d4c9c 2023-04-08 02:56:26,124 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/dd6f598366374a00a2c4f51fae2d4c9c, entries=150, sequenceid=298, filesize=11.9 K 2023-04-08 02:56:26,124 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/42a545fd2fc94998a0da12cea2797844 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/42a545fd2fc94998a0da12cea2797844 2023-04-08 02:56:26,128 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/42a545fd2fc94998a0da12cea2797844, entries=150, sequenceid=298, filesize=11.9 K 2023-04-08 02:56:26,128 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=127.47 KB/130530 for 337c881227bdfa873bb568857a6d7e4b in 444ms, sequenceid=298, compaction requested=true 2023-04-08 02:56:26,128 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:26,128 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:26,128 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:26,128 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 02:56:26,128 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 02:56:26,128 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:26,130 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 130382 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 02:56:26,130 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58646 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 02:56:26,130 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 337c881227bdfa873bb568857a6d7e4b/A is initiating minor compaction (all files) 2023-04-08 02:56:26,130 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 337c881227bdfa873bb568857a6d7e4b/B is initiating minor compaction (all files) 2023-04-08 02:56:26,130 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 337c881227bdfa873bb568857a6d7e4b/A in TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:26,130 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 337c881227bdfa873bb568857a6d7e4b/B in TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:26,130 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/7ae31d6bc02141edbbd2ad92db56764c, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/12e56920ca58499c84a41c069b51f1f4, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/6816d59a1f704876b767774761f45c26, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/f6a99ba0e8f446888fed4a9cd8b08c88, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/360a984c7d6e4db19c326cf4f62a1c9c] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp, totalSize=127.3 K 2023-04-08 02:56:26,130 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/dbb6ea05f4f34015aa28c353d6e34347, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/16b5beffcd694d7ea1679b6d773f305d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/688d702440754c58a86acd68cec74032, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/bade4b55b34a4875a26af36be0514c73, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/dd6f598366374a00a2c4f51fae2d4c9c] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp, totalSize=57.3 K 2023-04-08 02:56:26,131 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 7ae31d6bc02141edbbd2ad92db56764c, keycount=150, bloomtype=ROW, size=25.7 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680922582152 2023-04-08 02:56:26,131 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting dbb6ea05f4f34015aa28c353d6e34347, keycount=150, bloomtype=ROW, size=12.2 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680922582152 2023-04-08 02:56:26,131 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 12e56920ca58499c84a41c069b51f1f4, keycount=100, bloomtype=ROW, size=18.5 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680922583017 2023-04-08 02:56:26,131 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 16b5beffcd694d7ea1679b6d773f305d, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680922583017 2023-04-08 02:56:26,131 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 6816d59a1f704876b767774761f45c26, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1680922584190 2023-04-08 02:56:26,131 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 688d702440754c58a86acd68cec74032, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1680922584190 2023-04-08 02:56:26,131 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting f6a99ba0e8f446888fed4a9cd8b08c88, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1680922585339 2023-04-08 02:56:26,132 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting bade4b55b34a4875a26af36be0514c73, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1680922585339 2023-04-08 02:56:26,132 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 360a984c7d6e4db19c326cf4f62a1c9c, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1680922585674 2023-04-08 02:56:26,132 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting dd6f598366374a00a2c4f51fae2d4c9c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1680922585674 2023-04-08 02:56:26,143 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 337c881227bdfa873bb568857a6d7e4b#B#compaction#625 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:26,148 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 337c881227bdfa873bb568857a6d7e4b#A#compaction#626 average throughput is 4.93 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:26,151 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/a9f10714b9b1436cbb71a132c56ed159 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/a9f10714b9b1436cbb71a132c56ed159 2023-04-08 02:56:26,156 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 337c881227bdfa873bb568857a6d7e4b/B of 337c881227bdfa873bb568857a6d7e4b into a9f10714b9b1436cbb71a132c56ed159(size=12.5 K), total size for store is 12.5 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:26,156 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:26,156 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., storeName=337c881227bdfa873bb568857a6d7e4b/B, priority=11, startTime=1680922586128; duration=0sec 2023-04-08 02:56:26,156 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:26,156 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 02:56:26,158 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 58612 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 02:56:26,158 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 337c881227bdfa873bb568857a6d7e4b/C is initiating minor compaction (all files) 2023-04-08 02:56:26,158 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 337c881227bdfa873bb568857a6d7e4b/C in TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:26,158 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/558b7d7471224163a60dd4e6e30a34eb, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/14cb6dab080b41cba5f417d7faf51f00, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/51524096644e44bbb5d3095f857ed383, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/350c2a97fe214687ba7668954a74e352, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/42a545fd2fc94998a0da12cea2797844] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp, totalSize=57.2 K 2023-04-08 02:56:26,158 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 558b7d7471224163a60dd4e6e30a34eb, keycount=150, bloomtype=ROW, size=12.1 K, encoding=NONE, compression=NONE, seqNum=227, earliestPutTs=1680922582152 2023-04-08 02:56:26,159 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 14cb6dab080b41cba5f417d7faf51f00, keycount=100, bloomtype=ROW, size=9.4 K, encoding=NONE, compression=NONE, seqNum=237, earliestPutTs=1680922583017 2023-04-08 02:56:26,159 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 51524096644e44bbb5d3095f857ed383, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=277, earliestPutTs=1680922584190 2023-04-08 02:56:26,160 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 350c2a97fe214687ba7668954a74e352, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=287, earliestPutTs=1680922585339 2023-04-08 02:56:26,160 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 42a545fd2fc94998a0da12cea2797844, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1680922585674 2023-04-08 02:56:26,162 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/3af7315233b4426bbec6e82a4f53ffc0 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/3af7315233b4426bbec6e82a4f53ffc0 2023-04-08 02:56:26,166 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 337c881227bdfa873bb568857a6d7e4b/A of 337c881227bdfa873bb568857a6d7e4b into 3af7315233b4426bbec6e82a4f53ffc0(size=26.0 K), total size for store is 26.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:26,166 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:26,166 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:26,166 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., storeName=337c881227bdfa873bb568857a6d7e4b/A, priority=11, startTime=1680922586128; duration=0sec 2023-04-08 02:56:26,166 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:26,166 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:26,167 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 337c881227bdfa873bb568857a6d7e4b#C#compaction#627 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:26,168 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:26,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:26,169 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:26,170 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:26,170 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:26,170 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:26,170 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:26,170 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,170 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:26,172 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,172 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:26,172 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:26,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:26,172 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,172 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:26,172 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:26,173 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:26,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:26,174 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:26,174 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:26,174 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:26,174 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:26,174 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:26,174 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/0720a564ca8e48eebadb9e059e331bd9 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/0720a564ca8e48eebadb9e059e331bd9 2023-04-08 02:56:26,174 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:26,174 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:26,174 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:26,174 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:26,174 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:26,175 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=127.47 KB heapSize=360.38 KB 2023-04-08 02:56:26,175 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:26,175 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:26,175 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:26,175 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:26,175 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:26,175 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:26,178 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 337c881227bdfa873bb568857a6d7e4b/C of 337c881227bdfa873bb568857a6d7e4b into 0720a564ca8e48eebadb9e059e331bd9(size=12.4 K), total size for store is 12.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:26,178 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:26,178 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., storeName=337c881227bdfa873bb568857a6d7e4b/C, priority=11, startTime=1680922586128; duration=0sec 2023-04-08 02:56:26,178 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:26,188 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408b27c1222a0384ad1a32cf58162e84a2f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408b27c1222a0384ad1a32cf58162e84a2f 2023-04-08 02:56:26,193 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=335, memsize=42.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/10a6fdb7c2104dfc9054772005677ac1 2023-04-08 02:56:26,243 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:26,243 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:26,255 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:26,255 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 102 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680922646253, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,256 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:26,256 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 101 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922646255, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,299 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:26,299 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 113 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46618 deadline: 1680922646297, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,300 DEBUG [Thread-4170] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4143 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:56:26,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:26,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 111 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46610 deadline: 1680922646351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,353 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:26,353 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 115 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46594 deadline: 1680922646351, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,357 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:26,357 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 104 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680922646356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,359 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:26,359 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 103 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922646357, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:26,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 106 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46582 deadline: 1680922646558, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,560 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:26,560 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 105 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922646560, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,600 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/9e93b512b141450bae610114097f33c1 2023-04-08 02:56:26,609 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=42.49 KB at sequenceid=335 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/05c6a06e18ac4a5fa6c074f361737086 2023-04-08 02:56:26,613 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/10a6fdb7c2104dfc9054772005677ac1 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/10a6fdb7c2104dfc9054772005677ac1 2023-04-08 02:56:26,617 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/10a6fdb7c2104dfc9054772005677ac1, entries=150, sequenceid=335, filesize=25.4 K 2023-04-08 02:56:26,618 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/9e93b512b141450bae610114097f33c1 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/9e93b512b141450bae610114097f33c1 2023-04-08 02:56:26,621 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/9e93b512b141450bae610114097f33c1, entries=150, sequenceid=335, filesize=11.9 K 2023-04-08 02:56:26,622 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/05c6a06e18ac4a5fa6c074f361737086 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/05c6a06e18ac4a5fa6c074f361737086 2023-04-08 02:56:26,625 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/05c6a06e18ac4a5fa6c074f361737086, entries=150, sequenceid=335, filesize=11.9 K 2023-04-08 02:56:26,626 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~127.47 KB/130530, heapSize ~360.33 KB/368976, currentSize=53.67 KB/54960 for 337c881227bdfa873bb568857a6d7e4b in 451ms, sequenceid=335, compaction requested=false 2023-04-08 02:56:26,626 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:26,626 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:26,626 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:26,626 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:26,626 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,626 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:26,626 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:26,628 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:26,628 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:26,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:26,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:26,629 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:26,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,630 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:26,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:26,630 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@50b6ac2a[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:26,630 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:26,631 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,631 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,633 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,633 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,633 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:26,633 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:26,633 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:26,633 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:26,633 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,634 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:26,634 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,634 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:26,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:26,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:26,635 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:26,635 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:26,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:26,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:26,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:26,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:26,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:26,636 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,637 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:26,637 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:26,637 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:26,637 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:26,637 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:26,637 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:26,637 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:26,638 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,639 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:26,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:26,639 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:26,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:26,639 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,639 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:26,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:26,639 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:26,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:26,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:26,640 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,641 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:26,641 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:26,642 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,643 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,643 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:26,643 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:26,643 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:26,643 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,643 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:26,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:26,643 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:26,643 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:26,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:26,643 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:26,644 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,644 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:26,644 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,644 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:26,644 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:26,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:26,644 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,644 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,744 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:26,744 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:26,746 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:26,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:26,747 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:26,747 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:26,747 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:26,747 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:26,747 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:26,748 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,748 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:26,750 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:26,750 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,750 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:26,750 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:26,751 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,751 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:26,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:26,751 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:26,751 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:26,752 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:26,752 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:26,752 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:26,752 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:26,752 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:26,752 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:26,752 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:26,752 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:26,752 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:26,752 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:26,752 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=53.67 KB heapSize=160.13 KB 2023-04-08 02:56:26,753 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:26,753 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:26,753 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:26,753 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:26,753 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:26,753 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:26,763 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408c870721823ab453c97700805ab10fcaa to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408c870721823ab453c97700805ab10fcaa 2023-04-08 02:56:26,767 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=347, memsize=17.9 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/2624809fe2894b9fbb5f804f8e5f440d 2023-04-08 02:56:26,782 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/64ee3ac00f054e36b15b4138781b88ca 2023-04-08 02:56:26,790 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=17.89 KB at sequenceid=347 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/2e2e344230f6425e9e87cb61a4fd2f81 2023-04-08 02:56:26,793 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/2624809fe2894b9fbb5f804f8e5f440d as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/2624809fe2894b9fbb5f804f8e5f440d 2023-04-08 02:56:26,797 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/2624809fe2894b9fbb5f804f8e5f440d, entries=100, sequenceid=347, filesize=18.6 K 2023-04-08 02:56:26,798 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/64ee3ac00f054e36b15b4138781b88ca as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/64ee3ac00f054e36b15b4138781b88ca 2023-04-08 02:56:26,801 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/64ee3ac00f054e36b15b4138781b88ca, entries=100, sequenceid=347, filesize=9.5 K 2023-04-08 02:56:26,802 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/2e2e344230f6425e9e87cb61a4fd2f81 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/2e2e344230f6425e9e87cb61a4fd2f81 2023-04-08 02:56:26,805 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/2e2e344230f6425e9e87cb61a4fd2f81, entries=100, sequenceid=347, filesize=9.5 K 2023-04-08 02:56:26,806 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~53.67 KB/54960, heapSize ~160.08 KB/163920, currentSize=0 B/0 for 337c881227bdfa873bb568857a6d7e4b in 54ms, sequenceid=347, compaction requested=true 2023-04-08 02:56:26,806 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:26,806 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:26,806 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:26,806 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:26,806 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,806 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:26,806 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:26,807 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,807 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:26,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:26,808 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:26,808 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:26,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:26,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:26,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:26,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:26,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@547519d0[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:26,810 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:26,810 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,811 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,811 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,811 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,811 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:26,811 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:26,811 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:26,811 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,811 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:26,812 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,812 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:26,812 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:26,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:26,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:26,812 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:26,812 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:26,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:26,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:26,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:26,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:26,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:26,815 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:26,815 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:26,815 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:26,815 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:26,815 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:26,815 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:26,817 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,817 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:26,817 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:26,817 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:26,817 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:26,818 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:26,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:26,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:26,819 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:26,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:26,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,821 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,821 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:26,821 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:26,821 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:26,821 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:26,821 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:26,821 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:26,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:26,821 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:26,821 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:26,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:26,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:26,822 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,822 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:26,822 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,822 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:26,822 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:26,822 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,822 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,891 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:26,891 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-08 02:56:26,891 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:26,891 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:26,891 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:26,891 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:26,891 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:26,892 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:26,917 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408beed71896a1c49ceb04253821875e855 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408beed71896a1c49ceb04253821875e855 2023-04-08 02:56:26,922 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=357, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/c0ba1ed5554f4cb187f24aad6fa752bd 2023-04-08 02:56:26,922 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:26,922 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:26,924 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:26,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922586923, max timestamp=1680922586929Num uniques 859; 2023-04-08 02:56:26,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:56:26,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922586923, max timestamp=1680922586929Num uniques 859; 2023-04-08 02:56:26,930 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:56:26,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922586923, max timestamp=1680922586928Num uniques 859; 2023-04-08 02:56:26,931 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:56:26,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:26,932 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:26,932 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/7284cc7dbc06481f90bc175ee8a0d91e 2023-04-08 02:56:26,932 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:26,932 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:26,932 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:26,932 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:26,933 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,933 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:26,934 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:26,934 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,934 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:26,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:26,934 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,934 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:26,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:26,935 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:26,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:26,936 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:26,936 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:26,936 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:26,936 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:26,936 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:26,936 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:26,936 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:26,936 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:26,936 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:26,936 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:26,937 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:26,937 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:26,937 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:26,937 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,937 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:26,937 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:26,942 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,942 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:26,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:26,942 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,943 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:26,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:26,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:26,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:26,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:26,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:26,944 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@39898d41[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:26,944 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:26,944 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,945 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,945 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,946 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:26,946 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:26,946 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:26,946 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,946 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:26,947 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=357 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/dcf1503d625540dca4c4eb9f70d53178 2023-04-08 02:56:26,947 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:26,947 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,947 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:26,947 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:26,947 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,948 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:26,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:26,949 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:26,950 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:26,950 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:26,950 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:26,950 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:26,950 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:26,950 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:26,951 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/c0ba1ed5554f4cb187f24aad6fa752bd as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/c0ba1ed5554f4cb187f24aad6fa752bd 2023-04-08 02:56:26,952 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,952 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:26,952 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:26,952 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,952 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:26,953 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,953 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:26,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:26,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 117 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922646950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,953 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:26,953 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 128 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922646950, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680922586940, max timestamp=1680922586952Num uniques 1018; 2023-04-08 02:56:26,954 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-08 02:56:26,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680922586940, max timestamp=1680922586952Num uniques 1018; 2023-04-08 02:56:26,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-08 02:56:26,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=218, cellSize=9978, totalHeapSize=22912, min timestamp=1680922586940, max timestamp=1680922586952Num uniques 1018; 2023-04-08 02:56:26,955 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=9978, suffix heap size=133840,new segment heap size=22912 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=218 2023-04-08 02:56:26,955 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:26,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:26,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:26,957 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:26,957 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/c0ba1ed5554f4cb187f24aad6fa752bd, entries=200, sequenceid=357, filesize=32.3 K 2023-04-08 02:56:26,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,958 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/7284cc7dbc06481f90bc175ee8a0d91e as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/7284cc7dbc06481f90bc175ee8a0d91e 2023-04-08 02:56:26,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:26,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,961 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,961 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:26,961 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:26,961 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:26,961 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:26,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:26,961 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:26,961 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:26,961 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:26,961 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,961 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:26,962 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:26,961 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:26,962 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/7284cc7dbc06481f90bc175ee8a0d91e, entries=150, sequenceid=357, filesize=11.9 K 2023-04-08 02:56:26,962 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:26,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:26,962 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:26,962 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:26,962 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:26,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:26,962 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/dcf1503d625540dca4c4eb9f70d53178 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/dcf1503d625540dca4c4eb9f70d53178 2023-04-08 02:56:26,965 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/dcf1503d625540dca4c4eb9f70d53178, entries=150, sequenceid=357, filesize=11.9 K 2023-04-08 02:56:26,966 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=67.09 KB/68700 for 337c881227bdfa873bb568857a6d7e4b in 75ms, sequenceid=357, compaction requested=true 2023-04-08 02:56:26,966 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:26,966 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:26,966 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:26,966 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:56:26,966 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:56:26,966 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:26,967 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=73.80 KB heapSize=200.16 KB 2023-04-08 02:56:26,967 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:26,967 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:26,967 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:26,967 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:26,967 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:26,967 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:26,966 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:26,968 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 104756 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:56:26,968 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 337c881227bdfa873bb568857a6d7e4b/A is initiating minor compaction (all files) 2023-04-08 02:56:26,968 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46880 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:56:26,968 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 337c881227bdfa873bb568857a6d7e4b/A in TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:26,968 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 337c881227bdfa873bb568857a6d7e4b/B is initiating minor compaction (all files) 2023-04-08 02:56:26,968 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/3af7315233b4426bbec6e82a4f53ffc0, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/10a6fdb7c2104dfc9054772005677ac1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/2624809fe2894b9fbb5f804f8e5f440d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/c0ba1ed5554f4cb187f24aad6fa752bd] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp, totalSize=102.3 K 2023-04-08 02:56:26,968 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 337c881227bdfa873bb568857a6d7e4b/B in TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:26,969 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/a9f10714b9b1436cbb71a132c56ed159, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/9e93b512b141450bae610114097f33c1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/64ee3ac00f054e36b15b4138781b88ca, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/7284cc7dbc06481f90bc175ee8a0d91e] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp, totalSize=45.8 K 2023-04-08 02:56:26,969 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 3af7315233b4426bbec6e82a4f53ffc0, keycount=150, bloomtype=ROW, size=26.0 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1680922585674 2023-04-08 02:56:26,969 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting a9f10714b9b1436cbb71a132c56ed159, keycount=150, bloomtype=ROW, size=12.5 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1680922585674 2023-04-08 02:56:26,969 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 10a6fdb7c2104dfc9054772005677ac1, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1680922585736 2023-04-08 02:56:26,969 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 9e93b512b141450bae610114097f33c1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1680922585736 2023-04-08 02:56:26,969 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 2624809fe2894b9fbb5f804f8e5f440d, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1680922586252 2023-04-08 02:56:26,970 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 64ee3ac00f054e36b15b4138781b88ca, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1680922586252 2023-04-08 02:56:26,970 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting c0ba1ed5554f4cb187f24aad6fa752bd, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1680922586864 2023-04-08 02:56:26,970 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 7284cc7dbc06481f90bc175ee8a0d91e, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1680922586883 2023-04-08 02:56:26,978 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 337c881227bdfa873bb568857a6d7e4b#B#compaction#638 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:26,980 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 337c881227bdfa873bb568857a6d7e4b#A#compaction#639 average throughput is 6.57 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:26,989 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/eb3c96c14b8c466094def0a464eef38e as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/eb3c96c14b8c466094def0a464eef38e 2023-04-08 02:56:26,993 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 337c881227bdfa873bb568857a6d7e4b/B of 337c881227bdfa873bb568857a6d7e4b into eb3c96c14b8c466094def0a464eef38e(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:26,993 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:26,993 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., storeName=337c881227bdfa873bb568857a6d7e4b/B, priority=12, startTime=1680922586966; duration=0sec 2023-04-08 02:56:26,993 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:26,993 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:56:26,994 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 46846 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:56:26,994 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 337c881227bdfa873bb568857a6d7e4b/C is initiating minor compaction (all files) 2023-04-08 02:56:26,994 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 337c881227bdfa873bb568857a6d7e4b/C in TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:26,994 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/0720a564ca8e48eebadb9e059e331bd9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/05c6a06e18ac4a5fa6c074f361737086, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/2e2e344230f6425e9e87cb61a4fd2f81, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/dcf1503d625540dca4c4eb9f70d53178] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp, totalSize=45.7 K 2023-04-08 02:56:26,995 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 0720a564ca8e48eebadb9e059e331bd9, keycount=150, bloomtype=ROW, size=12.4 K, encoding=NONE, compression=NONE, seqNum=298, earliestPutTs=1680922585674 2023-04-08 02:56:26,995 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 05c6a06e18ac4a5fa6c074f361737086, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=335, earliestPutTs=1680922585736 2023-04-08 02:56:26,996 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 2e2e344230f6425e9e87cb61a4fd2f81, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=347, earliestPutTs=1680922586252 2023-04-08 02:56:26,996 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting dcf1503d625540dca4c4eb9f70d53178, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1680922586883 2023-04-08 02:56:26,999 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/96785c1e3ab042adbddddd50def28a14 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/96785c1e3ab042adbddddd50def28a14 2023-04-08 02:56:27,003 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 337c881227bdfa873bb568857a6d7e4b#C#compaction#640 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:27,003 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 337c881227bdfa873bb568857a6d7e4b/A of 337c881227bdfa873bb568857a6d7e4b into 96785c1e3ab042adbddddd50def28a14(size=26.1 K), total size for store is 26.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:27,003 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:27,003 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., storeName=337c881227bdfa873bb568857a6d7e4b/A, priority=12, startTime=1680922586966; duration=0sec 2023-04-08 02:56:27,003 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:27,008 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:27,008 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680922647004, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,009 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:27,009 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46582 deadline: 1680922647005, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,055 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:27,055 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 119 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922647054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,057 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:27,057 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 130 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922647054, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,062 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:27,062 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:27,064 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:27,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:27,065 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:27,065 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:27,065 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:27,065 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:27,065 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:27,065 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,065 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,067 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:27,067 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,067 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:27,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:27,067 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,067 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:27,067 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,067 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:27,068 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,068 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:27,068 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:27,068 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:27,068 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:27,068 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:27,068 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:27,068 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:27,068 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:27,068 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:27,068 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:27,069 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:27,069 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:27,069 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:27,069 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:27,069 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,069 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:27,069 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:27,070 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,070 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:27,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:27,070 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,071 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:27,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:27,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:27,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,071 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:27,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:27,072 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@288be47f[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:27,072 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:27,072 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,073 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,073 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,073 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,073 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:27,073 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:27,073 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:27,073 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,074 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:27,076 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:27,076 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:27,076 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,076 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:27,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:27,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:27,076 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:27,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:27,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,077 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:27,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,078 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:27,079 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:27,079 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:27,079 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:27,079 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:27,079 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:27,079 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:27,080 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,080 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:27,080 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:27,080 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,080 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:27,081 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,081 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:27,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:27,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:27,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,081 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,082 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:27,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,084 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:27,084 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,084 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,084 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,084 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,084 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,084 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,084 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:27,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:27,084 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:27,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:27,085 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:27,084 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:27,084 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,084 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:27,085 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:27,085 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:27,085 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:27,085 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:27,110 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:27,110 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680922647109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,111 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:27,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46582 deadline: 1680922647109, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,185 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:27,186 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:27,187 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:27,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:27,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:27,189 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:27,189 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:27,189 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:27,189 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:27,189 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,189 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,190 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,191 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:27,191 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:27,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:27,191 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,191 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:27,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,191 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:27,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,192 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:27,192 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:27,192 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:27,192 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:27,192 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:27,192 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:27,192 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:27,192 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:27,192 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:27,192 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:27,192 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:27,192 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:27,192 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:27,192 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:27,192 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,193 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:27,193 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:27,194 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,194 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,194 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:27,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:27,195 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,195 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:27,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:27,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:27,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:27,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:27,196 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4d39dab8[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:27,196 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:27,196 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,198 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,198 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,198 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:27,198 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:27,198 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:27,198 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,198 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:27,199 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:27,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:27,199 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:27,199 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:27,199 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:27,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,200 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:27,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,201 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:27,201 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:27,201 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:27,201 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:27,201 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:27,202 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:27,202 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:27,203 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,203 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:27,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:27,204 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:27,204 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:27,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:27,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:27,204 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,205 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:27,205 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,206 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:27,206 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,207 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,209 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,209 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:27,209 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,209 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:27,209 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:27,209 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,209 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,209 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,209 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,209 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:27,209 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:27,209 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:27,209 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:27,210 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:27,210 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:27,210 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:27,210 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:27,257 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:27,258 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 121 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922647256, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,260 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:27,260 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 132 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922647258, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,310 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:27,310 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:27,312 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:27,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:27,313 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:27,313 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:27,313 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:27,313 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:27,313 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:27,313 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,313 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:27,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680922647311, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,314 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:27,314 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46582 deadline: 1680922647312, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,314 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,314 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:27,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:27,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:27,315 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,315 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:27,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:27,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,315 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:27,316 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:27,316 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:27,316 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:27,316 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:27,316 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:27,316 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:27,316 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:27,316 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:27,316 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:27,316 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:27,316 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:27,316 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:27,316 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:27,316 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,316 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:27,316 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:27,318 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,318 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:27,318 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,318 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:27,318 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,319 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:27,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:27,319 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@317ee156[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:27,319 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:27,319 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,324 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,324 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,324 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,325 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:27,325 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:27,325 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:27,326 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,326 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:27,326 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,326 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:27,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:27,326 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:27,326 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:27,326 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:27,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:27,327 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:27,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:27,328 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,329 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:27,329 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:27,329 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:27,329 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:27,329 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:27,329 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:27,329 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:27,331 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,331 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:27,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,331 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:27,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:27,331 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:27,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:27,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:27,331 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:27,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,333 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:27,333 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,335 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,335 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,335 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,335 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:27,335 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:27,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:27,335 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:27,335 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:27,335 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,335 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:27,335 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:27,335 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:27,335 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,335 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:27,335 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,335 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:27,336 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,336 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:27,336 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:27,378 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304087e723c6d792742dfa6e5b7e9b18ababd to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304087e723c6d792742dfa6e5b7e9b18ababd 2023-04-08 02:56:27,410 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/0c711a7cd18e4c99a3fa13aca596d652 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/0c711a7cd18e4c99a3fa13aca596d652 2023-04-08 02:56:27,414 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 337c881227bdfa873bb568857a6d7e4b/C of 337c881227bdfa873bb568857a6d7e4b into 0c711a7cd18e4c99a3fa13aca596d652(size=12.6 K), total size for store is 12.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:27,414 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:27,414 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., storeName=337c881227bdfa873bb568857a6d7e4b/C, priority=12, startTime=1680922586966; duration=0sec 2023-04-08 02:56:27,414 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:27,436 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:27,436 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:27,438 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:27,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:27,440 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:27,440 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:27,440 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:27,440 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:27,440 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:27,441 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,441 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,443 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:27,443 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,443 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:27,443 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,443 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:27,443 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,443 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:27,444 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,444 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:27,444 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:27,444 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:27,444 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:27,444 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:27,444 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:27,444 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:27,444 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:27,444 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:27,444 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:27,444 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:27,444 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:27,444 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:27,444 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:27,445 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,445 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:27,445 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:27,446 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:27,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:27,446 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,446 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:27,446 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,447 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:27,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:27,447 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@446881bd[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:27,448 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:27,448 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,449 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,449 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,449 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,449 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:27,449 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:27,449 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:27,449 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,449 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:27,450 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,450 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:27,450 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:27,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:27,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:27,450 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:27,450 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,451 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:27,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,452 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:27,452 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:27,452 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:27,452 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:27,452 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:27,452 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:27,452 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:27,459 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,459 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,459 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:27,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:27,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:27,459 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:27,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:27,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:27,459 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,459 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:27,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,460 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:27,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,462 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,463 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:27,463 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,463 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:27,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:27,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:27,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:27,463 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:27,463 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:27,463 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,463 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,463 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:27,463 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:27,463 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,464 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:27,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:27,464 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,464 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,464 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,561 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:27,561 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 123 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46656 deadline: 1680922647561, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,564 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:27,564 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:27,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 134 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922647563, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,564 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:27,566 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:27,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:27,567 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:27,567 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:27,567 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:27,567 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:27,567 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:27,567 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,567 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,568 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,568 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:27,569 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:27,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:27,569 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,569 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:27,569 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,569 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:27,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,570 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:27,570 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:27,570 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:27,570 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:27,570 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:27,570 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:27,570 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:27,570 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:27,570 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:27,570 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:27,570 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:27,571 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:27,571 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:27,571 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:27,571 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,571 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:27,571 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:27,572 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,572 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:27,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:27,572 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,572 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:27,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:27,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:27,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:27,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:27,574 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@33372d53[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:27,574 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:27,574 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,576 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,576 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,576 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,577 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:27,577 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:27,577 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:27,577 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,577 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:27,578 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,578 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:27,578 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:27,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:27,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:27,578 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:27,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:27,578 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:27,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:27,579 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,580 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:27,580 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:27,580 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:27,580 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:27,580 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:27,580 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:27,580 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:27,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:27,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:27,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:27,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:27,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:27,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,583 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:27,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,584 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,584 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:27,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,586 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:27,587 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:27,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:27,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:27,587 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,587 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,587 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,587 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:27,587 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:27,587 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:27,587 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:27,587 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:27,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:27,587 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,587 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:27,587 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:27,587 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,587 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,616 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:27,616 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46582 deadline: 1680922647615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,629 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:27,629 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 141 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680922647628, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,687 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:27,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:27,690 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:27,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:27,692 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:27,692 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:27,692 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:27,692 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:27,692 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:27,693 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,693 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,695 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:27,695 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:27,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:27,695 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,696 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,696 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:27,696 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,696 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:27,697 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,697 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:27,700 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:27,700 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:27,700 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:27,700 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:27,700 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:27,701 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:27,701 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:27,701 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:27,701 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:27,701 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:27,701 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:27,701 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:27,701 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:27,701 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,701 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:27,701 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:27,703 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,703 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:27,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:27,703 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:27,704 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,704 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:27,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:27,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,705 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,706 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:27,706 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:27,707 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@30d8dd97[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:27,707 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:27,707 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,709 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,709 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,709 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:27,709 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:27,709 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:27,709 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,709 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:27,718 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:27,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:27,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:27,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:27,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:27,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,722 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:27,722 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:27,722 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:27,722 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:27,722 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:27,722 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:27,722 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:27,723 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:27,723 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:27,723 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:27,724 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,724 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:27,724 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,724 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:27,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:27,725 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:27,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:27,726 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:27,729 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:27,729 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,730 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,730 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:27,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:27,731 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,733 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:27,733 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:27,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:27,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:27,733 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,733 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,733 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,733 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,733 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,733 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:27,733 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,733 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:27,733 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:27,733 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:27,733 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:27,734 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:27,734 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:27,734 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:27,783 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=401, memsize=24.6 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/47cceee4253d4eab93e0df0a2adee5b2 2023-04-08 02:56:27,807 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/c35cb67497d24afba40d9e8da20b6d60 2023-04-08 02:56:27,821 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=24.60 KB at sequenceid=401 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/cd13504f803c473484f1bbb32605048f 2023-04-08 02:56:27,826 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/47cceee4253d4eab93e0df0a2adee5b2 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/47cceee4253d4eab93e0df0a2adee5b2 2023-04-08 02:56:27,834 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:27,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:27,836 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:27,843 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/47cceee4253d4eab93e0df0a2adee5b2, entries=250, sequenceid=401, filesize=39.2 K 2023-04-08 02:56:27,846 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/c35cb67497d24afba40d9e8da20b6d60 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/c35cb67497d24afba40d9e8da20b6d60 2023-04-08 02:56:27,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:27,848 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:27,848 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:27,848 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:27,848 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:27,848 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:27,849 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,849 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,850 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:27,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:27,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:27,850 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,851 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:27,851 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:27,851 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:27,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:27,852 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:27,852 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:27,864 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:27,864 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:27,864 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:27,864 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:27,864 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:27,866 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/c35cb67497d24afba40d9e8da20b6d60, entries=150, sequenceid=401, filesize=11.9 K 2023-04-08 02:56:27,867 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/cd13504f803c473484f1bbb32605048f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/cd13504f803c473484f1bbb32605048f 2023-04-08 02:56:27,871 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/cd13504f803c473484f1bbb32605048f, entries=150, sequenceid=401, filesize=11.9 K 2023-04-08 02:56:27,871 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~73.80 KB/75570, heapSize ~200.11 KB/204912, currentSize=107.34 KB/109920 for 337c881227bdfa873bb568857a6d7e4b in 904ms, sequenceid=401, compaction requested=false 2023-04-08 02:56:27,871 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:27,872 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:27,872 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:27,872 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:27,872 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:27,872 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=107.34 KB heapSize=319.50 KB 2023-04-08 02:56:27,872 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:27,872 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:27,872 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:27,872 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:27,872 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:27,872 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:27,911 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040854a91fb8b8fd4a1c9038296b4dff0239 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040854a91fb8b8fd4a1c9038296b4dff0239 2023-04-08 02:56:28,066 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:28,066 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:28,078 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:28,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922648076, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:28,079 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:28,079 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46656 deadline: 1680922648078, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:28,121 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:28,121 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46582 deadline: 1680922648119, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:28,136 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:28,136 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 143 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680922648135, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:28,180 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:28,180 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922648179, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:28,181 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:28,181 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46656 deadline: 1680922648180, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:28,316 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=423, memsize=35.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/2485e3c30ba047b2b60cacacc9d29b7d 2023-04-08 02:56:28,338 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=423 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/65a6b8c0cbaa4f748d82ac909abfe8be 2023-04-08 02:56:28,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:28,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922648382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:28,383 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:28,383 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 135 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46656 deadline: 1680922648382, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:28,686 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:28,686 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 137 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46656 deadline: 1680922648684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:28,687 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:28,687 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922648684, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:28,760 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=35.78 KB at sequenceid=423 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/bbf62a9b766a4b05bd9507489a83f635 2023-04-08 02:56:28,764 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/2485e3c30ba047b2b60cacacc9d29b7d as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/2485e3c30ba047b2b60cacacc9d29b7d 2023-04-08 02:56:28,769 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/2485e3c30ba047b2b60cacacc9d29b7d, entries=150, sequenceid=423, filesize=25.4 K 2023-04-08 02:56:28,770 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/65a6b8c0cbaa4f748d82ac909abfe8be as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/65a6b8c0cbaa4f748d82ac909abfe8be 2023-04-08 02:56:28,773 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/65a6b8c0cbaa4f748d82ac909abfe8be, entries=150, sequenceid=423, filesize=11.9 K 2023-04-08 02:56:28,776 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/bbf62a9b766a4b05bd9507489a83f635 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/bbf62a9b766a4b05bd9507489a83f635 2023-04-08 02:56:28,790 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/bbf62a9b766a4b05bd9507489a83f635, entries=150, sequenceid=423, filesize=11.9 K 2023-04-08 02:56:28,791 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~107.34 KB/109920, heapSize ~319.45 KB/327120, currentSize=67.09 KB/68700 for 337c881227bdfa873bb568857a6d7e4b in 919ms, sequenceid=423, compaction requested=true 2023-04-08 02:56:28,791 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:28,791 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:28,791 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:28,791 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:28,791 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:28,791 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:28,791 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:28,794 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:28,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:28,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:28,794 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:28,794 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:28,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:28,795 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:28,795 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:28,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:28,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:28,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:28,796 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:28,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:28,796 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@766afe8b[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:28,796 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:28,796 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:28,797 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:28,797 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:28,797 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:28,797 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:28,797 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:28,797 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:28,797 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:28,797 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:28,798 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:28,798 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:28,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:28,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:28,799 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:28,799 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:28,799 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:28,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:28,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:28,799 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:28,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:28,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:28,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:28,800 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:28,801 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:28,801 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:28,801 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:28,801 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:28,801 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:28,801 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:28,801 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:28,803 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:28,803 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:28,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:28,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:28,803 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:28,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:28,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:28,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:28,803 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:28,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:28,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:28,803 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:28,804 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:28,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:28,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:28,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:28,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:28,805 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:28,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:28,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:28,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:28,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:28,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:28,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:28,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:28,808 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:28,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:28,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:28,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:28,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:28,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:28,808 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:28,808 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:28,808 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:28,808 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:28,808 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:28,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:28,809 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:28,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:28,909 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:28,910 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:28,911 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:28,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:28,912 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:28,912 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:28,912 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:28,912 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:28,912 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:28,913 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:28,913 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:28,914 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:28,914 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:28,914 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:28,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:28,914 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:28,915 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:28,915 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:28,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:28,915 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:28,915 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:28,915 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:28,916 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:28,916 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:28,916 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:28,916 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:28,916 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:28,916 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:28,916 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:28,916 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=67.09 KB heapSize=199.97 KB 2023-04-08 02:56:28,916 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:28,916 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:28,916 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:28,916 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:28,916 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:28,916 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:28,928 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408fea0f27d30ea4ecab7879d4aa78f91e0 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408fea0f27d30ea4ecab7879d4aa78f91e0 2023-04-08 02:56:28,932 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=436, memsize=22.4 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/6cfa536c55594bdabb40010912698049 2023-04-08 02:56:28,940 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/eecaed1b39d44b8da8bb2601cac402dc 2023-04-08 02:56:29,132 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:29,132 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:29,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:29,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680922649165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,167 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:29,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 149 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46582 deadline: 1680922649165, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,189 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:29,189 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 139 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46656 deadline: 1680922649188, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,191 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:29,191 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 148 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922649189, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:29,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680922649268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,270 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:29,270 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 151 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46582 deadline: 1680922649268, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,349 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=22.36 KB at sequenceid=436 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/c5c5655b56914dd68f201e66a098118a 2023-04-08 02:56:29,353 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/6cfa536c55594bdabb40010912698049 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/6cfa536c55594bdabb40010912698049 2023-04-08 02:56:29,356 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/6cfa536c55594bdabb40010912698049, entries=150, sequenceid=436, filesize=25.4 K 2023-04-08 02:56:29,356 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/eecaed1b39d44b8da8bb2601cac402dc as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/eecaed1b39d44b8da8bb2601cac402dc 2023-04-08 02:56:29,359 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/eecaed1b39d44b8da8bb2601cac402dc, entries=150, sequenceid=436, filesize=11.9 K 2023-04-08 02:56:29,360 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/c5c5655b56914dd68f201e66a098118a as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/c5c5655b56914dd68f201e66a098118a 2023-04-08 02:56:29,363 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/c5c5655b56914dd68f201e66a098118a, entries=150, sequenceid=436, filesize=11.9 K 2023-04-08 02:56:29,364 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~67.09 KB/68700, heapSize ~199.92 KB/204720, currentSize=114.05 KB/116790 for 337c881227bdfa873bb568857a6d7e4b in 448ms, sequenceid=436, compaction requested=true 2023-04-08 02:56:29,364 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:29,364 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:29,364 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:29,364 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:29,364 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,364 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:29,364 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:29,366 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:29,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:29,366 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:29,366 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,366 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:29,366 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:29,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:29,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:29,367 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@768fec92[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:29,367 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:29,368 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,369 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,369 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,369 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:29,369 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:29,369 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:29,369 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,369 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:29,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:29,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:29,372 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,372 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:29,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:29,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:29,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:29,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:29,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:29,373 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,374 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:29,374 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:29,374 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:29,374 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:29,374 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:29,374 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:29,374 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:29,376 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,386 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,386 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:29,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:29,386 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:29,386 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:29,387 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,387 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:29,388 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,388 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,389 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,390 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,390 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:29,390 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:29,390 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,390 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:29,390 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:29,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:29,390 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:29,390 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,390 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,390 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:29,390 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:29,390 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:29,390 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,391 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:29,391 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:29,391 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:29,391 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:29,391 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:29,473 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:29,473 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922589157, max timestamp=1680922589472Num uniques 859; 2023-04-08 02:56:29,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922589157, max timestamp=1680922589472Num uniques 859; 2023-04-08 02:56:29,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922589157, max timestamp=1680922589472Num uniques 859; 2023-04-08 02:56:29,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:56:29,474 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=120.76 KB heapSize=360.05 KB 2023-04-08 02:56:29,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:56:29,474 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:56:29,475 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:29,475 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:29,475 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:29,475 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:29,475 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:29,475 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:29,490 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408221d61dbdfca45518613025137a4a541 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408221d61dbdfca45518613025137a4a541 2023-04-08 02:56:29,491 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:29,491 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:29,493 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:29,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:29,495 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:29,495 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:29,495 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:29,495 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:29,495 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:29,495 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,495 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:29,496 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,496 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:29,497 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:29,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:29,497 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,497 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:29,497 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:29,497 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:29,498 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:29,498 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:29,498 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:29,498 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:29,498 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:29,498 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:29,498 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:29,498 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:29,498 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:29,499 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:29,499 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:29,499 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:29,499 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:29,499 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:29,499 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,499 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:29,499 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:29,502 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,502 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:29,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:29,502 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,502 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:29,502 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:29,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:29,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,503 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:29,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:29,504 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c580aa5[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:29,504 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:29,504 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,505 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,506 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,506 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,506 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:29,506 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:29,506 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:29,506 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,506 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:29,507 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,507 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:29,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:29,507 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:29,507 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:29,507 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:29,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:29,508 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:29,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,509 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,510 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,511 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:29,511 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:29,511 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:29,511 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:29,511 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:29,511 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:29,511 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:29,512 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,512 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:29,512 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:29,513 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:29,513 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,513 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,514 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,515 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,515 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922589498, max timestamp=1680922589514Num uniques 859; 2023-04-08 02:56:29,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:29,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922589498, max timestamp=1680922589514Num uniques 859; 2023-04-08 02:56:29,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:56:29,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:56:29,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922589498, max timestamp=1680922589514Num uniques 859; 2023-04-08 02:56:29,516 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:56:29,516 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,521 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:29,521 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,521 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:29,522 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:29,521 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:29,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:29,522 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:29,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:29,522 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:29,521 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:29,522 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:29,522 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,522 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:29,522 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:29,522 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,522 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,522 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,522 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:29,522 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:29,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680922589553, max timestamp=1680922589558Num uniques 1018; 2023-04-08 02:56:29,559 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-08 02:56:29,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680922589553, max timestamp=1680922589558Num uniques 1018; 2023-04-08 02:56:29,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680922589553, max timestamp=1680922589558Num uniques 1018; 2023-04-08 02:56:29,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-08 02:56:29,560 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-08 02:56:29,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680922589581, max timestamp=1680922589589Num uniques 1027; 2023-04-08 02:56:29,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-08 02:56:29,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680922589581, max timestamp=1680922589589Num uniques 1027; 2023-04-08 02:56:29,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-08 02:56:29,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=177, cellSize=8102, totalHeapSize=18648, min timestamp=1680922589581, max timestamp=1680922589589Num uniques 1027; 2023-04-08 02:56:29,591 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47032, new segment data size=8102, suffix heap size=134776,new segment heap size=18648 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1027, new segment cells count=177 2023-04-08 02:56:29,622 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:29,623 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:29,624 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:29,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:29,625 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:29,625 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:29,625 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:29,626 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:29,626 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:29,626 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,626 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:29,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680922589613, max timestamp=1680922589629Num uniques 1036; 2023-04-08 02:56:29,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-08 02:56:29,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680922589613, max timestamp=1680922589629Num uniques 1036; 2023-04-08 02:56:29,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=186, cellSize=8516, totalHeapSize=19584, min timestamp=1680922589613, max timestamp=1680922589629Num uniques 1036; 2023-04-08 02:56:29,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-08 02:56:29,633 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=47446, new segment data size=8516, suffix heap size=135712,new segment heap size=19584 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1036, new segment cells count=186 2023-04-08 02:56:29,633 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:29,633 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,633 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:29,633 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:29,634 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,634 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:29,634 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:29,634 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:29,635 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:29,635 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:29,640 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:29,640 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:29,640 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:29,640 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:29,640 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:29,640 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:29,640 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:29,640 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:29,640 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:29,640 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:29,640 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:29,641 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:29,641 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:29,641 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,641 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:29,641 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:29,642 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,642 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:29,643 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:29,643 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:29,643 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:29,644 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,645 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:29,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:29,645 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@a9a44dc[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:29,645 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:29,645 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,647 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,647 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,647 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,647 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:29,647 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:29,647 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:29,647 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,647 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:29,655 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,655 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:29,655 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:29,655 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:29,655 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:29,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,656 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:29,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,657 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:29,657 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:29,657 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:29,657 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:29,658 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:29,658 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:29,658 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:29,660 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,660 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:29,660 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:29,660 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:29,660 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,661 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:29,662 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,662 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,663 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:29,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,664 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:29,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,665 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,665 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,669 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,669 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:29,670 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:29,670 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:29,670 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:29,670 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:29,670 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:29,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:29,670 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:29,670 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:29,670 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,670 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,670 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,670 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,670 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:29,670 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:29,671 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:29,671 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:29,677 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680922589651, max timestamp=1680922589675Num uniques 1045; 2023-04-08 02:56:29,677 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-08 02:56:29,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680922589651, max timestamp=1680922589675Num uniques 1045; 2023-04-08 02:56:29,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=195, cellSize=8930, totalHeapSize=20520, min timestamp=1680922589651, max timestamp=1680922589675Num uniques 1045; 2023-04-08 02:56:29,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-08 02:56:29,678 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47860, new segment data size=8930, suffix heap size=136648,new segment heap size=20520 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1045, new segment cells count=195 2023-04-08 02:56:29,711 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680922589697, max timestamp=1680922589709Num uniques 1054; 2023-04-08 02:56:29,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680922589697, max timestamp=1680922589709Num uniques 1054; 2023-04-08 02:56:29,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=204, cellSize=9340, totalHeapSize=21456, min timestamp=1680922589697, max timestamp=1680922589709Num uniques 1054; 2023-04-08 02:56:29,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-08 02:56:29,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-08 02:56:29,712 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=48270, new segment data size=9340, suffix heap size=137584,new segment heap size=21456 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1054, new segment cells count=204 2023-04-08 02:56:29,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680922589723, max timestamp=1680922589746Num uniques 1063; 2023-04-08 02:56:29,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-08 02:56:29,747 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680922589723, max timestamp=1680922589746Num uniques 1063; 2023-04-08 02:56:29,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-08 02:56:29,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=163, cellSize=7458, totalHeapSize=17192, min timestamp=1680922589723, max timestamp=1680922589746Num uniques 1063; 2023-04-08 02:56:29,748 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=48678, new segment data size=7458, suffix heap size=138520,new segment heap size=17192 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1063, new segment cells count=163 2023-04-08 02:56:29,771 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:29,771 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:29,772 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:29,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:29,774 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:29,774 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:29,774 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:29,774 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:29,774 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:29,774 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,774 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:29,775 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,775 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:29,775 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:29,775 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:29,776 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,776 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:29,776 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:29,776 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:29,777 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:29,777 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:29,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680922589766, max timestamp=1680922589776Num uniques 1022; 2023-04-08 02:56:29,778 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-08 02:56:29,778 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:29,778 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:29,778 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:29,778 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:29,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680922589766, max timestamp=1680922589776Num uniques 1022; 2023-04-08 02:56:29,779 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:29,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-08 02:56:29,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=172, cellSize=7872, totalHeapSize=18128, min timestamp=1680922589766, max timestamp=1680922589776Num uniques 1022; 2023-04-08 02:56:29,779 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:29,779 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:29,779 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:29,779 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=46802, new segment data size=7872, suffix heap size=134256,new segment heap size=18128 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1022, new segment cells count=172 2023-04-08 02:56:29,779 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:29,779 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:29,779 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:29,779 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:29,779 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:29,779 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,779 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:29,779 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:29,782 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,782 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:29,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:29,782 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,782 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:29,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:29,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:29,782 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:29,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:29,783 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2ba57da2[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:29,783 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:29,783 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,784 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,784 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,784 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,784 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:29,784 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:29,784 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:29,784 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,784 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:29,785 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:29,785 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,785 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:29,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:29,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:29,786 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:29,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:29,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:29,786 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:29,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,787 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,788 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:29,788 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:29,788 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:29,788 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:29,788 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:29,788 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:29,788 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:29,789 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,789 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:29,789 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,789 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:29,789 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:29,790 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:29,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:29,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:29,790 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,790 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:29,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,791 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,791 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:29,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,792 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,794 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,794 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:29,794 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:29,794 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:29,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:29,794 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:29,794 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,794 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:29,794 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:29,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:29,795 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:29,794 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:29,795 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:29,795 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,795 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,795 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:29,795 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:29,815 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680922589802, max timestamp=1680922589813Num uniques 1031; 2023-04-08 02:56:29,815 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=231, cellSize=10576, totalHeapSize=24264, min timestamp=1680922589802, max timestamp=1680922589813Num uniques 1031; 2023-04-08 02:56:29,815 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-08 02:56:29,815 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=10576, suffix heap size=135192,new segment heap size=24264 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=231 2023-04-08 02:56:29,816 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=181, cellSize=8286, totalHeapSize=19064, min timestamp=1680922589802, max timestamp=1680922589813Num uniques 1031; 2023-04-08 02:56:29,816 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=47216, new segment data size=8286, suffix heap size=135192,new segment heap size=19064 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1031, new segment cells count=181 2023-04-08 02:56:29,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680922589849, max timestamp=1680922589863Num uniques 1040; 2023-04-08 02:56:29,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-08 02:56:29,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680922589849, max timestamp=1680922589863Num uniques 1090; 2023-04-08 02:56:29,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=49920, new segment data size=10990, suffix heap size=141328,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1090, new segment cells count=240 2023-04-08 02:56:29,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=240, cellSize=10990, totalHeapSize=25200, min timestamp=1680922589849, max timestamp=1680922589863Num uniques 1040; 2023-04-08 02:56:29,866 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=47630, new segment data size=10990, suffix heap size=136128,new segment heap size=25200 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1040, new segment cells count=240 2023-04-08 02:56:29,895 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:29,895 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:29,896 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=458, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/dd2591ebd6c44801a5566da8d1048c3b 2023-04-08 02:56:29,897 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:29,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:29,898 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:29,898 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:29,898 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:29,898 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:29,898 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:29,899 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,899 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:29,900 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,900 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:29,900 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:29,900 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,900 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:29,900 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:29,901 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:29,901 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:29,901 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:29,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:29,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:29,901 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:29,901 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:29,901 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:29,901 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:29,901 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:29,901 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:29,902 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:29,902 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:29,902 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:29,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:29,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:29,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:29,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:29,903 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,903 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:29,904 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,904 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:29,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:29,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:29,905 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2247ff38[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:29,905 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:29,906 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,907 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,907 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,907 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,907 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:29,907 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:29,907 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,907 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:29,907 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:29,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680922589898, max timestamp=1680922589907Num uniques 1099; 2023-04-08 02:56:29,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-08 02:56:29,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680922589898, max timestamp=1680922589907Num uniques 1099; 2023-04-08 02:56:29,909 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-08 02:56:29,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=199, cellSize=9114, totalHeapSize=20936, min timestamp=1680922589898, max timestamp=1680922589907Num uniques 1099; 2023-04-08 02:56:29,910 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=50334, new segment data size=9114, suffix heap size=142264,new segment heap size=20936 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1099, new segment cells count=199 2023-04-08 02:56:29,914 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:29,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:29,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:29,914 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:29,914 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:29,914 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:29,914 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:29,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:29,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:29,916 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:29,916 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:29,916 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:29,916 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:29,916 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:29,916 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:29,918 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,918 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:29,918 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,918 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:29,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:29,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:29,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:29,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:29,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,919 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:29,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,920 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:29,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:29,921 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,922 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,922 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:29,922 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:29,922 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:29,922 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:29,922 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:29,922 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:29,922 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:29,923 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:29,923 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:29,923 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:29,922 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:29,922 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:29,923 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:29,923 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:29,923 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:29,923 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,923 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:29,923 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:29,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680922589931, max timestamp=1680922589940Num uniques 1058; 2023-04-08 02:56:29,942 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-08 02:56:29,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680922589931, max timestamp=1680922589940Num uniques 1058; 2023-04-08 02:56:29,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=158, cellSize=7230, totalHeapSize=16672, min timestamp=1680922589931, max timestamp=1680922589940Num uniques 1058; 2023-04-08 02:56:29,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-08 02:56:29,943 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=48450, new segment data size=7230, suffix heap size=138000,new segment heap size=16672 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1058, new segment cells count=158 2023-04-08 02:56:29,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680922589962, max timestamp=1680922589986Num uniques 1017; 2023-04-08 02:56:29,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680922589962, max timestamp=1680922589986Num uniques 1017; 2023-04-08 02:56:29,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-04-08 02:56:29,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=167, cellSize=7642, totalHeapSize=17608, min timestamp=1680922589962, max timestamp=1680922589986Num uniques 1017; 2023-04-08 02:56:29,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-04-08 02:56:29,990 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46572, new segment data size=7642, suffix heap size=133736,new segment heap size=17608 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1017, new segment cells count=167 2023-04-08 02:56:30,023 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:30,024 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:30,026 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:30,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:30,027 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:30,027 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:30,027 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:30,027 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:30,027 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:30,028 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,028 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,030 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,030 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:30,030 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:30,030 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:30,030 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,031 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:30,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,031 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:30,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,031 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:30,032 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:30,032 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:30,032 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:30,032 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:30,032 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:30,032 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:30,032 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:30,033 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:30,033 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:30,033 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:30,033 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:30,033 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:30,033 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:30,033 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,033 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:30,033 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:30,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1680922590022, max timestamp=1680922590033Num uniques 1026; 2023-04-08 02:56:30,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1680922590022, max timestamp=1680922590033Num uniques 1026; 2023-04-08 02:56:30,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=10346, suffix heap size=134672,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=226 2023-04-08 02:56:30,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=226, cellSize=10346, totalHeapSize=23744, min timestamp=1680922590022, max timestamp=1680922590033Num uniques 1026; 2023-04-08 02:56:30,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=10346, suffix heap size=134672,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=226 2023-04-08 02:56:30,036 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=46986, new segment data size=10346, suffix heap size=134672,new segment heap size=23744 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1026, new segment cells count=226 2023-04-08 02:56:30,036 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,036 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:30,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:30,037 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:30,037 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:30,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:30,037 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:30,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:30,038 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1e76c4f7[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:30,038 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:30,038 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,039 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,039 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,039 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,039 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:30,040 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:30,040 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:30,040 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,040 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:30,041 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:30,041 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:30,041 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:30,041 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,041 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:30,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:30,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:30,042 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:30,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,043 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:30,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,044 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,045 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:30,045 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:30,045 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:30,045 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:30,045 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:30,045 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:30,045 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:30,050 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,051 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:30,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,051 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:30,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:30,051 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:30,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:30,051 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:30,052 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,052 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:30,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:30,054 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,056 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,056 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:30,056 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:30,056 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:30,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:30,056 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:30,056 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:30,056 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:30,056 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,056 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,057 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,057 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,057 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,057 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:30,057 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:30,057 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:30,057 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:30,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680922590057, max timestamp=1680922590085Num uniques 1085; 2023-04-08 02:56:30,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-04-08 02:56:30,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680922590057, max timestamp=1680922590085Num uniques 1085; 2023-04-08 02:56:30,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=235, cellSize=10760, totalHeapSize=24680, min timestamp=1680922590057, max timestamp=1680922590085Num uniques 1085; 2023-04-08 02:56:30,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-04-08 02:56:30,088 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=49690, new segment data size=10760, suffix heap size=140808,new segment heap size=24680 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1085, new segment cells count=235 2023-04-08 02:56:30,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1680922590114, max timestamp=1680922590140Num uniques 1094; 2023-04-08 02:56:30,143 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=11174, suffix heap size=141744,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=244 2023-04-08 02:56:30,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1680922590114, max timestamp=1680922590140Num uniques 1094; 2023-04-08 02:56:30,144 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=11174, suffix heap size=141744,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=244 2023-04-08 02:56:30,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=244, cellSize=11174, totalHeapSize=25616, min timestamp=1680922590114, max timestamp=1680922590140Num uniques 1094; 2023-04-08 02:56:30,145 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=50104, new segment data size=11174, suffix heap size=141744,new segment heap size=25616 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1094, new segment cells count=244 2023-04-08 02:56:30,157 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:30,157 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:30,159 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:30,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:30,165 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:30,165 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:30,165 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:30,165 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:30,165 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:30,166 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,166 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,167 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:30,167 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,167 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:30,167 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:30,168 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,168 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:30,168 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,168 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:30,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,169 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:30,172 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:30,172 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:30,172 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:30,172 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:30,172 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:30,180 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:30,180 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:30,180 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:30,180 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:30,180 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:30,180 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:30,180 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:30,180 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:30,180 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,180 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:30,181 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:30,182 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:30,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:30,182 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:30,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:30,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:30,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:30,184 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@4612cf41[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:30,184 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:30,184 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,186 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,187 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:30,187 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(162): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,187 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,187 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:30,187 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:30,187 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:30,187 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:30,189 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,189 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:30,189 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:30,189 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:30,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:30,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:30,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:30,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:30,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:30,191 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,193 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:30,193 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:30,193 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:30,193 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:30,193 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:30,193 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:30,193 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:30,195 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,195 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:30,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,195 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,195 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,195 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:30,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:30,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:30,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:30,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,196 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:30,196 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:30,197 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,198 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:30,198 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,199 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,202 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:30,202 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,202 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:30,202 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:30,202 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,202 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,202 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:30,202 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,202 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,202 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,202 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,202 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1680922590186, max timestamp=1680922590200Num uniques 1103; 2023-04-08 02:56:30,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=9295, suffix heap size=142680,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=203 2023-04-08 02:56:30,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1680922590186, max timestamp=1680922590200Num uniques 1103; 2023-04-08 02:56:30,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=203, cellSize=9295, totalHeapSize=21352, min timestamp=1680922590186, max timestamp=1680922590200Num uniques 1103; 2023-04-08 02:56:30,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=9295, suffix heap size=142680,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=203 2023-04-08 02:56:30,203 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=50515, new segment data size=9295, suffix heap size=142680,new segment heap size=21352 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1103, new segment cells count=203 2023-04-08 02:56:30,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:30,203 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:30,203 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:30,203 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:30,203 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:30,203 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:30,203 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:30,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1680922590224, max timestamp=1680922590234Num uniques 1062; 2023-04-08 02:56:30,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1680922590224, max timestamp=1680922590234Num uniques 1062; 2023-04-08 02:56:30,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-04-08 02:56:30,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=162, cellSize=7412, totalHeapSize=17088, min timestamp=1680922590224, max timestamp=1680922590234Num uniques 1062; 2023-04-08 02:56:30,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-04-08 02:56:30,238 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=48632, new segment data size=7412, suffix heap size=138416,new segment heap size=17088 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1062, new segment cells count=162 2023-04-08 02:56:30,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1680922590264, max timestamp=1680922590272Num uniques 1021; 2023-04-08 02:56:30,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1680922590264, max timestamp=1680922590272Num uniques 1021; 2023-04-08 02:56:30,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=10116, suffix heap size=134152,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=221 2023-04-08 02:56:30,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=221, cellSize=10116, totalHeapSize=23224, min timestamp=1680922590264, max timestamp=1680922590272Num uniques 1021; 2023-04-08 02:56:30,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=10116, suffix heap size=134152,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=221 2023-04-08 02:56:30,275 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46756, new segment data size=10116, suffix heap size=134152,new segment heap size=23224 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1021, new segment cells count=221 2023-04-08 02:56:30,304 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:30,304 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:30,306 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/58377c872ad44a15960cf8ee36c94085 2023-04-08 02:56:30,306 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:30,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1680922590298, max timestamp=1680922590306Num uniques 1080; 2023-04-08 02:56:30,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=12820, suffix heap size=140288,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=280 2023-04-08 02:56:30,309 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1680922590298, max timestamp=1680922590306Num uniques 1080; 2023-04-08 02:56:30,310 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=12820, suffix heap size=140288,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=280 2023-04-08 02:56:30,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=280, cellSize=12820, totalHeapSize=29360, min timestamp=1680922590298, max timestamp=1680922590306Num uniques 1080; 2023-04-08 02:56:30,311 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-8] regionserver.CompactionPipeline(163): Suffix data size=49460, new segment data size=12820, suffix heap size=140288,new segment heap size=29360 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1080, new segment cells count=280 2023-04-08 02:56:30,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:30,311 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:30,311 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:30,311 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:30,311 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:30,311 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:30,312 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,312 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,313 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,313 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:30,313 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:30,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:30,313 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,314 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:30,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,314 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:30,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,314 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:30,320 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:30,320 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:30,320 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:30,320 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:30,320 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:30,325 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:30,326 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:30,326 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:30,326 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=458 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/9044562768224551825fd54fa6d77df1 2023-04-08 02:56:30,326 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:30,328 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:30,328 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:30,328 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:30,328 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:30,328 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,328 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:30,328 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:30,330 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,330 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,330 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/dd2591ebd6c44801a5566da8d1048c3b as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/dd2591ebd6c44801a5566da8d1048c3b 2023-04-08 02:56:30,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:30,330 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:30,331 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,331 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:30,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:30,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:30,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,331 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,332 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:30,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:30,332 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@46dae24c[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:30,332 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:30,332 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,334 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,334 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,334 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,334 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:30,334 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:30,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 317 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680922650329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,334 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 318 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46582 deadline: 1680922650329, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1680922590325, max timestamp=1680922590332Num uniques 1139; 2023-04-08 02:56:30,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-6] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=10944, suffix heap size=146424,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=239 2023-04-08 02:56:30,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=150, cellSize=6870, totalHeapSize=15840, min timestamp=1680922590325, max timestamp=1680922590333Num uniques 1139; 2023-04-08 02:56:30,335 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=239, cellSize=10944, totalHeapSize=25096, min timestamp=1680922590325, max timestamp=1680922590332Num uniques 1139; 2023-04-08 02:56:30,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-1] regionserver.CompactionPipeline(163): Suffix data size=52164, new segment data size=10944, suffix heap size=146424,new segment heap size=25096 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=239 2023-04-08 02:56:30,336 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/dd2591ebd6c44801a5566da8d1048c3b, entries=150, sequenceid=458, filesize=25.4 K 2023-04-08 02:56:30,336 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-4] regionserver.CompactionPipeline(163): Suffix data size=52154, new segment data size=6870, suffix heap size=146424,new segment heap size=15840 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1139, new segment cells count=150 2023-04-08 02:56:30,336 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:30,336 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,337 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:30,337 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:30,337 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:30,337 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/58377c872ad44a15960cf8ee36c94085 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/58377c872ad44a15960cf8ee36c94085 2023-04-08 02:56:30,339 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:30,339 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:30,339 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:30,339 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:30,339 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:30,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:30,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:30,340 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,341 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/58377c872ad44a15960cf8ee36c94085, entries=150, sequenceid=458, filesize=11.9 K 2023-04-08 02:56:30,341 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:30,342 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/9044562768224551825fd54fa6d77df1 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/9044562768224551825fd54fa6d77df1 2023-04-08 02:56:30,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,342 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,343 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:30,343 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:30,343 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:30,343 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:30,343 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:30,343 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:30,343 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:30,345 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:30,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:30,345 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,345 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:30,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,345 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:30,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:30,345 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:30,346 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,346 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:30,346 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/9044562768224551825fd54fa6d77df1, entries=150, sequenceid=458, filesize=11.9 K 2023-04-08 02:56:30,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,347 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~86.11 KB/88176, currentSize=76.54 KB/78376 for 337c881227bdfa873bb568857a6d7e4b in 873ms, sequenceid=458, compaction requested=true 2023-04-08 02:56:30,347 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:30,347 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:30,347 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:30,347 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 02:56:30,347 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 02:56:30,347 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:30,347 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,347 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:30,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,348 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,349 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61646 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 02:56:30,349 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 337c881227bdfa873bb568857a6d7e4b/B is initiating minor compaction (all files) 2023-04-08 02:56:30,349 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 337c881227bdfa873bb568857a6d7e4b/B in TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:30,349 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/eb3c96c14b8c466094def0a464eef38e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/c35cb67497d24afba40d9e8da20b6d60, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/65a6b8c0cbaa4f748d82ac909abfe8be, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/eecaed1b39d44b8da8bb2601cac402dc, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/58377c872ad44a15960cf8ee36c94085] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp, totalSize=60.2 K 2023-04-08 02:56:30,349 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 145030 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 02:56:30,349 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 337c881227bdfa873bb568857a6d7e4b/A is initiating minor compaction (all files) 2023-04-08 02:56:30,349 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting eb3c96c14b8c466094def0a464eef38e, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1680922586883 2023-04-08 02:56:30,349 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 337c881227bdfa873bb568857a6d7e4b/A in TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:30,349 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/96785c1e3ab042adbddddd50def28a14, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/47cceee4253d4eab93e0df0a2adee5b2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/2485e3c30ba047b2b60cacacc9d29b7d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/6cfa536c55594bdabb40010912698049, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/dd2591ebd6c44801a5566da8d1048c3b] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp, totalSize=141.6 K 2023-04-08 02:56:30,350 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting c35cb67497d24afba40d9e8da20b6d60, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1680922586952 2023-04-08 02:56:30,350 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:30,350 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:30,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:30,350 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 65a6b8c0cbaa4f748d82ac909abfe8be, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=423, earliestPutTs=1680922587000 2023-04-08 02:56:30,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:30,350 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 96785c1e3ab042adbddddd50def28a14, keycount=150, bloomtype=ROW, size=26.1 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1680922586883 2023-04-08 02:56:30,350 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,350 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:30,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:30,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:30,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:30,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:30,350 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,350 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 47cceee4253d4eab93e0df0a2adee5b2, keycount=250, bloomtype=ROW, size=39.2 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1680922586952 2023-04-08 02:56:30,350 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting eecaed1b39d44b8da8bb2601cac402dc, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1680922588073 2023-04-08 02:56:30,350 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:30,351 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,351 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,351 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,351 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,351 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 2485e3c30ba047b2b60cacacc9d29b7d, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=423, earliestPutTs=1680922587000 2023-04-08 02:56:30,351 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,351 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:30,351 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:30,351 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 58377c872ad44a15960cf8ee36c94085, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1680922589157 2023-04-08 02:56:30,351 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 6cfa536c55594bdabb40010912698049, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1680922588073 2023-04-08 02:56:30,352 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:30,352 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=89.96 KB heapSize=248.91 KB 2023-04-08 02:56:30,352 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting dd2591ebd6c44801a5566da8d1048c3b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1680922589157 2023-04-08 02:56:30,353 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:30,353 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:30,353 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:30,354 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:30,354 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:30,354 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:30,361 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 337c881227bdfa873bb568857a6d7e4b#B#compaction#653 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:30,374 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 337c881227bdfa873bb568857a6d7e4b#A#compaction#654 average throughput is 1.41 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:30,378 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:30,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46656 deadline: 1680922650375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,378 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:30,378 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 174 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922650375, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,382 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:30,382 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 127 service: ClientService methodName: Mutate size: 4.6 K connection: 172.31.14.131:46618 deadline: 1680922650378, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,384 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408bd060db7b4384a2eb84c5b42abcfe309 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408bd060db7b4384a2eb84c5b42abcfe309 2023-04-08 02:56:30,388 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/7fed4017f811488896b29200b5150fe6 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/7fed4017f811488896b29200b5150fe6 2023-04-08 02:56:30,391 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 337c881227bdfa873bb568857a6d7e4b/B of 337c881227bdfa873bb568857a6d7e4b into 7fed4017f811488896b29200b5150fe6(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:30,392 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:30,392 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., storeName=337c881227bdfa873bb568857a6d7e4b/B, priority=11, startTime=1680922590347; duration=0sec 2023-04-08 02:56:30,392 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:30,392 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 02:56:30,394 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=832, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/c4a9e0f392ba47b9b7bc3582907915c6 2023-04-08 02:56:30,395 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61612 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 02:56:30,395 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 337c881227bdfa873bb568857a6d7e4b/C is initiating minor compaction (all files) 2023-04-08 02:56:30,395 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 337c881227bdfa873bb568857a6d7e4b/C in TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:30,395 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/0c711a7cd18e4c99a3fa13aca596d652, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/cd13504f803c473484f1bbb32605048f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/bbf62a9b766a4b05bd9507489a83f635, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/c5c5655b56914dd68f201e66a098118a, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/9044562768224551825fd54fa6d77df1] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp, totalSize=60.2 K 2023-04-08 02:56:30,396 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 0c711a7cd18e4c99a3fa13aca596d652, keycount=150, bloomtype=ROW, size=12.6 K, encoding=NONE, compression=NONE, seqNum=357, earliestPutTs=1680922586883 2023-04-08 02:56:30,396 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting cd13504f803c473484f1bbb32605048f, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=401, earliestPutTs=1680922586952 2023-04-08 02:56:30,397 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting bbf62a9b766a4b05bd9507489a83f635, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=423, earliestPutTs=1680922587000 2023-04-08 02:56:30,397 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting c5c5655b56914dd68f201e66a098118a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=436, earliestPutTs=1680922588073 2023-04-08 02:56:30,397 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 9044562768224551825fd54fa6d77df1, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1680922589157 2023-04-08 02:56:30,404 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/a54a66c1701f4d94b46554322126fa6b as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/a54a66c1701f4d94b46554322126fa6b 2023-04-08 02:56:30,408 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 337c881227bdfa873bb568857a6d7e4b#C#compaction#656 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:30,408 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 337c881227bdfa873bb568857a6d7e4b/A of 337c881227bdfa873bb568857a6d7e4b into a54a66c1701f4d94b46554322126fa6b(size=26.3 K), total size for store is 26.3 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:30,408 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:30,408 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., storeName=337c881227bdfa873bb568857a6d7e4b/A, priority=11, startTime=1680922590347; duration=0sec 2023-04-08 02:56:30,408 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:30,437 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:30,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 319 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680922650436, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,438 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:30,438 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 320 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46582 deadline: 1680922650437, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,451 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:30,452 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:30,453 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:30,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:30,454 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:30,454 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:30,454 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:30,454 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:30,454 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:30,454 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,455 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,460 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:30,460 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,460 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:30,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:30,460 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,460 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:30,460 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,460 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:30,461 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,461 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:30,461 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:30,461 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:30,461 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:30,461 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:30,461 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:30,461 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:30,461 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:30,461 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:30,461 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:30,461 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:30,461 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:30,462 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:30,462 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:30,462 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,462 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:30,462 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:30,463 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,463 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:30,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:30,463 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,463 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:30,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:30,463 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:30,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,464 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:30,464 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:30,465 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5bf97b90[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:30,465 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:30,465 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,466 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,466 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,466 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,466 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:30,466 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,466 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:30,466 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:30,466 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:30,467 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,467 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:30,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:30,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:30,467 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:30,467 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:30,467 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:30,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:30,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:30,468 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,469 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:30,469 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:30,469 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:30,469 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:30,469 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:30,469 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:30,469 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:30,472 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,472 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:30,472 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:30,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:30,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,472 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:30,472 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:30,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:30,472 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,473 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:30,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,474 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,474 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:30,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,475 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,476 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,476 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,476 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,476 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,476 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:30,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:30,476 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:30,476 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:30,476 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:30,476 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,476 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,477 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:30,476 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:30,477 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:30,477 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:30,477 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:30,477 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:30,477 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:30,482 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:30,482 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:30,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46656 deadline: 1680922650479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,482 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 176 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922650479, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,485 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:30,485 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 129 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46618 deadline: 1680922650483, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,577 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:30,578 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:30,579 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:30,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:30,580 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:30,580 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:30,580 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:30,580 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:30,580 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:30,581 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,581 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,582 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:30,582 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,582 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:30,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:30,582 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,582 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:30,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,583 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:30,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,583 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:30,583 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:30,583 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:30,583 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:30,583 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:30,583 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:30,584 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:30,584 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:30,584 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:30,584 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:30,584 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:30,584 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:30,584 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:30,584 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:30,584 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,584 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:30,584 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:30,585 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,585 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:30,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:30,586 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,586 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:30,586 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,587 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:30,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:30,587 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@73805d64[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:30,587 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:30,587 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,589 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,589 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,589 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,589 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:30,589 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:30,589 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:30,589 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,589 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:30,590 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,590 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:30,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:30,590 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:30,590 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:30,591 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:30,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:30,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:30,591 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:30,592 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:30,593 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:30,593 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:30,593 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:30,593 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:30,593 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:30,593 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:30,596 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,596 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:30,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,596 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,596 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:30,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:30,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:30,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:30,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:30,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,597 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,597 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:30,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,598 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:30,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,601 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,601 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,601 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:30,601 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:30,601 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:30,601 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,601 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,601 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:30,601 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:30,601 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,602 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,605 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:30,605 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:30,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:30,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:30,605 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:30,605 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:30,642 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:30,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 321 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680922650640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,643 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:30,643 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 322 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46582 deadline: 1680922650640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:30,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46656 deadline: 1680922650683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,684 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:30,684 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922650683, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,688 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:30,688 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 131 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46618 deadline: 1680922650687, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,702 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:30,702 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:30,703 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:30,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:30,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:30,704 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:30,704 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:30,704 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:30,704 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:30,705 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,705 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,707 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:30,707 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,707 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:30,707 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:30,708 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,708 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:30,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,708 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:30,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,708 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:30,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:30,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:30,709 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:30,709 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:30,709 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:30,709 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:30,709 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:30,709 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:30,709 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:30,709 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:30,709 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:30,709 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:30,709 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:30,709 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,709 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:30,709 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:30,711 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,711 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:30,711 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,711 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:30,711 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:30,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:30,712 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2338fc92[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:30,712 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:30,712 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,713 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,713 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,713 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:30,713 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:30,713 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:30,714 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,714 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:30,714 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:30,714 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:30,715 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:30,715 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:30,715 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:30,717 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:30,717 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:30,717 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:30,717 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:30,717 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:30,717 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:30,719 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,719 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:30,719 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:30,719 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,719 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:30,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:30,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,721 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:30,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,721 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,722 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,723 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,723 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,723 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:30,723 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:30,723 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,723 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,723 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,723 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:30,723 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:30,723 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:30,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:30,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:30,723 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:30,723 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:30,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:30,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:30,805 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=832 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/47c44a7c021e40c0a99c830f82143464 2023-04-08 02:56:30,814 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=29.58 KB at sequenceid=832 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/c0319156bd0542f08054e1121fa3882b 2023-04-08 02:56:30,818 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/c4a9e0f392ba47b9b7bc3582907915c6 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/c4a9e0f392ba47b9b7bc3582907915c6 2023-04-08 02:56:30,821 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/18d1b2d2eaa54600bcd657749a006d4b as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/18d1b2d2eaa54600bcd657749a006d4b 2023-04-08 02:56:30,822 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/c4a9e0f392ba47b9b7bc3582907915c6, entries=200, sequenceid=832, filesize=32.3 K 2023-04-08 02:56:30,823 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/47c44a7c021e40c0a99c830f82143464 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/47c44a7c021e40c0a99c830f82143464 2023-04-08 02:56:30,823 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:30,824 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:30,825 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:30,826 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 337c881227bdfa873bb568857a6d7e4b/C of 337c881227bdfa873bb568857a6d7e4b into 18d1b2d2eaa54600bcd657749a006d4b(size=12.7 K), total size for store is 12.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:30,826 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:30,826 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., storeName=337c881227bdfa873bb568857a6d7e4b/C, priority=11, startTime=1680922590347; duration=0sec 2023-04-08 02:56:30,826 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:30,827 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/47c44a7c021e40c0a99c830f82143464, entries=150, sequenceid=832, filesize=11.9 K 2023-04-08 02:56:30,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:30,827 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:30,827 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:30,827 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:30,827 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:30,827 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:30,828 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/c0319156bd0542f08054e1121fa3882b as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/c0319156bd0542f08054e1121fa3882b 2023-04-08 02:56:30,828 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,828 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,829 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:30,829 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,829 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:30,829 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:30,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,830 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,830 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:30,830 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:30,830 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,831 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:30,831 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:30,831 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:30,831 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:30,831 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:30,831 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:30,831 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/c0319156bd0542f08054e1121fa3882b, entries=150, sequenceid=832, filesize=11.9 K 2023-04-08 02:56:30,831 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:30,831 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:30,832 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:30,832 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:30,831 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:30,832 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:30,832 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~96.67 KB/98986, heapSize ~268.79 KB/275240, currentSize=87.22 KB/89310 for 337c881227bdfa873bb568857a6d7e4b in 480ms, sequenceid=832, compaction requested=false 2023-04-08 02:56:30,832 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:30,832 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:30,832 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:30,832 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,832 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:30,832 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:30,833 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,834 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,834 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:30,834 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:30,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:30,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:30,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:30,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,835 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,836 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:30,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:30,836 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@32e1a23b[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:30,836 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:30,836 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,841 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,841 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,841 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,841 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:30,841 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:30,841 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:30,841 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,841 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:30,842 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,842 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:30,842 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:30,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,842 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:30,842 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:30,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:30,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:30,845 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:30,845 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:30,845 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:30,845 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:30,845 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:30,846 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:30,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:30,846 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,846 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:30,846 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,846 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:30,847 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,848 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:30,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,850 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,850 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,850 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,850 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,850 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,850 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,850 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,850 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:30,850 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:30,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:30,850 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:30,850 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:30,850 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:30,850 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:30,850 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:30,851 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:30,851 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:30,851 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:30,851 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:30,945 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:30,945 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-08 02:56:30,945 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:30,946 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:30,946 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:30,946 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:30,946 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:30,946 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:30,951 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:30,951 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:30,953 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:30,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:30,955 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:30,955 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:30,955 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:30,955 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:30,955 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:30,955 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,955 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,957 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:30,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:30,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:30,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,958 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:30,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,959 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:30,959 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:30,959 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:30,959 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:30,960 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:30,960 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:30,960 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:30,960 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:30,960 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:30,962 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,962 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,962 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:30,964 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,964 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,964 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:30,964 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a08fcf8[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:30,964 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:30,964 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:30,965 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,967 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,967 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:30,967 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:30,967 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:30,967 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,968 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:30,969 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,969 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:30,969 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:30,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:30,969 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:30,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:30,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:30,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 329 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680922650967, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:30,970 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:30,970 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46582 deadline: 1680922650968, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:30,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:30,974 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:30,974 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:30,974 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:30,974 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:30,974 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:30,974 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:30,978 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:30,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:30,979 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,979 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:30,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,980 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:30,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:30,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,981 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:30,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:30,982 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,982 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,985 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:30,985 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:30,985 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,985 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:30,985 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:30,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:30,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:30,985 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:30,986 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,986 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,986 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:30,986 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:30,985 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:30,986 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:30,986 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:30,986 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:30,986 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:30,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:30,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46656 deadline: 1680922650985, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,987 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:30,987 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922650986, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:30,991 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:30,991 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 133 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46618 deadline: 1680922650989, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:31,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46582 deadline: 1680922651071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,073 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:31,073 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 331 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680922651071, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,086 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:31,087 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:31,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:31,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:31,089 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:31,089 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:31,089 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:31,089 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:31,089 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:31,090 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,090 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:31,091 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:31,091 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,091 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:31,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:31,091 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,091 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:31,091 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:31,091 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:31,092 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:31,092 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:31,092 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:31,092 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:31,092 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:31,092 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:31,092 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:31,092 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:31,092 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:31,092 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:31,093 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:31,093 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:31,093 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:31,093 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:31,093 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:31,093 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,093 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:31,093 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:31,094 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,094 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:31,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:31,094 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,094 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:31,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:31,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:31,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:31,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,096 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:31,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:31,096 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@22546a0c[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:31,096 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:31,096 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,098 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,098 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,098 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,098 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,098 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:31,098 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:31,098 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:31,099 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:31,100 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,100 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:31,100 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:31,100 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:31,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:31,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:31,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:31,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:31,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:31,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:31,102 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:31,102 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:31,102 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:31,102 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:31,102 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:31,104 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,104 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:31,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:31,104 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:31,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,104 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:31,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:31,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:31,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:31,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:31,105 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:31,105 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:31,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:31,107 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,108 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:31,108 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:31,108 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:31,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:31,108 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:31,108 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,108 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,108 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,108 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,108 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:31,108 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:31,108 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:31,108 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:31,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:31,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:31,108 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:31,108 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:31,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:31,208 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:31,209 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:31,210 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:31,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:31,211 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:31,211 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:31,212 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:31,212 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:31,212 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:31,214 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,214 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:31,216 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,216 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:31,216 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:31,216 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:31,217 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,217 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:31,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:31,217 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:31,217 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:31,217 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:31,218 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:31,218 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:31,218 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:31,218 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:31,218 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:31,218 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:31,218 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:31,218 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:31,218 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:31,218 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:31,218 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:31,218 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:31,218 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:31,218 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,218 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:31,218 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:31,220 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,220 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:31,220 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,220 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:31,220 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:31,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:31,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,221 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:31,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:31,222 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@534e785f[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:31,222 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:31,222 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,223 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,223 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,223 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,223 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:31,223 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:31,223 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:31,223 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,223 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:31,228 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,228 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:31,228 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:31,228 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:31,228 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,229 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,230 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:31,230 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:31,230 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:31,231 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:31,231 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:31,231 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:31,231 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:31,232 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,232 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,232 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:31,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:31,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,232 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:31,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:31,233 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:31,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:31,233 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:31,233 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,234 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,234 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:31,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:31,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,236 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,236 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:31,236 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:31,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:31,236 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:31,237 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:31,236 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:31,236 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:31,237 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:31,236 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:31,237 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:31,237 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,237 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,237 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,237 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,244 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:31,244 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:31,244 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:31,244 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:31,276 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:31,276 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46582 deadline: 1680922651274, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,276 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:31,277 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 333 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680922651275, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,344 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:31,345 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:31,346 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:31,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:31,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:31,347 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:31,347 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:31,347 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:31,347 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:31,348 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,348 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:31,350 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,350 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:31,350 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:31,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:31,350 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,350 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:31,350 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:31,350 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:31,351 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:31,351 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:31,351 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:31,351 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:31,351 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:31,351 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:31,351 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:31,351 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:31,351 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:31,351 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:31,351 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:31,352 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:31,352 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:31,352 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:31,352 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:31,352 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,352 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:31,352 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:31,353 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,353 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:31,353 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:31,354 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,354 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:31,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:31,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:31,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:31,354 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,355 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:31,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:31,355 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@37f6c2c8[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:31,355 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:31,355 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,356 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,356 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,356 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,356 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:31,357 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,357 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:31,357 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:31,357 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:31,358 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:31,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:31,358 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:31,358 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:31,358 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:31,358 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:31,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:31,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:31,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:31,359 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:31,360 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,361 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:31,361 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:31,361 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:31,361 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:31,361 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:31,361 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:31,361 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:31,363 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040878f1b83d618c4f12ae2afed0f086a860 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040878f1b83d618c4f12ae2afed0f086a860 2023-04-08 02:56:31,367 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,367 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:31,367 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,367 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:31,367 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:31,368 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=852, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/a146727b257547aead830f6b888f312b 2023-04-08 02:56:31,368 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:31,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:31,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:31,368 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,369 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:31,369 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:31,370 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:31,371 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,371 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:31,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:31,372 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,374 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,374 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:31,374 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:31,374 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:31,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:31,374 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:31,374 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:31,374 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:31,374 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:31,374 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:31,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:31,374 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:31,375 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,375 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,375 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,375 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,375 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:31,375 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:31,377 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=852 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/447aee9245c84aa1bd79851be90bec5a 2023-04-08 02:56:31,384 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=852 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/327448e0c7fc43528e927a2b21699d0a 2023-04-08 02:56:31,388 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/a146727b257547aead830f6b888f312b as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/a146727b257547aead830f6b888f312b 2023-04-08 02:56:31,391 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/a146727b257547aead830f6b888f312b, entries=200, sequenceid=852, filesize=32.3 K 2023-04-08 02:56:31,392 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/447aee9245c84aa1bd79851be90bec5a as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/447aee9245c84aa1bd79851be90bec5a 2023-04-08 02:56:31,395 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/447aee9245c84aa1bd79851be90bec5a, entries=150, sequenceid=852, filesize=11.9 K 2023-04-08 02:56:31,395 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/327448e0c7fc43528e927a2b21699d0a as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/327448e0c7fc43528e927a2b21699d0a 2023-04-08 02:56:31,398 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/327448e0c7fc43528e927a2b21699d0a, entries=150, sequenceid=852, filesize=11.9 K 2023-04-08 02:56:31,398 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 337c881227bdfa873bb568857a6d7e4b in 453ms, sequenceid=852, compaction requested=true 2023-04-08 02:56:31,398 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:31,399 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:31,399 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:56:31,399 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:56:31,399 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:31,399 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:31,399 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 93104 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:56:31,400 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 337c881227bdfa873bb568857a6d7e4b/A is initiating minor compaction (all files) 2023-04-08 02:56:31,400 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37444 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:56:31,400 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 337c881227bdfa873bb568857a6d7e4b/A in TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:31,400 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 337c881227bdfa873bb568857a6d7e4b/B is initiating minor compaction (all files) 2023-04-08 02:56:31,400 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/a54a66c1701f4d94b46554322126fa6b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/c4a9e0f392ba47b9b7bc3582907915c6, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/a146727b257547aead830f6b888f312b] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp, totalSize=90.9 K 2023-04-08 02:56:31,400 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 337c881227bdfa873bb568857a6d7e4b/B in TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:31,400 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/7fed4017f811488896b29200b5150fe6, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/47c44a7c021e40c0a99c830f82143464, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/447aee9245c84aa1bd79851be90bec5a] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp, totalSize=36.6 K 2023-04-08 02:56:31,400 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting a54a66c1701f4d94b46554322126fa6b, keycount=150, bloomtype=ROW, size=26.3 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1680922589157 2023-04-08 02:56:31,400 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 7fed4017f811488896b29200b5150fe6, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1680922589157 2023-04-08 02:56:31,401 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting c4a9e0f392ba47b9b7bc3582907915c6, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=832, earliestPutTs=1680922590344 2023-04-08 02:56:31,401 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 47c44a7c021e40c0a99c830f82143464, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=832, earliestPutTs=1680922590344 2023-04-08 02:56:31,401 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting a146727b257547aead830f6b888f312b, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=852, earliestPutTs=1680922590369 2023-04-08 02:56:31,402 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 447aee9245c84aa1bd79851be90bec5a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=852, earliestPutTs=1680922590373 2023-04-08 02:56:31,406 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 337c881227bdfa873bb568857a6d7e4b#B#compaction#662 average throughput is unlimited, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:31,413 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 337c881227bdfa873bb568857a6d7e4b#A#compaction#661 average throughput is 2.82 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:31,426 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/c7f8ce256513403a9958bf3fae4c413e as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/c7f8ce256513403a9958bf3fae4c413e 2023-04-08 02:56:31,430 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 337c881227bdfa873bb568857a6d7e4b/B of 337c881227bdfa873bb568857a6d7e4b into c7f8ce256513403a9958bf3fae4c413e(size=12.9 K), total size for store is 12.9 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:31,430 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:31,430 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., storeName=337c881227bdfa873bb568857a6d7e4b/B, priority=13, startTime=1680922591399; duration=0sec 2023-04-08 02:56:31,430 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:31,430 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 3 store files, 0 compacting, 3 eligible, 16 blocking 2023-04-08 02:56:31,431 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 3 files of size 37410 starting at candidate #0 after considering 1 permutations with 1 in ratio 2023-04-08 02:56:31,431 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 337c881227bdfa873bb568857a6d7e4b/C is initiating minor compaction (all files) 2023-04-08 02:56:31,431 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 337c881227bdfa873bb568857a6d7e4b/C in TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:31,431 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/18d1b2d2eaa54600bcd657749a006d4b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/c0319156bd0542f08054e1121fa3882b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/327448e0c7fc43528e927a2b21699d0a] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp, totalSize=36.5 K 2023-04-08 02:56:31,432 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 18d1b2d2eaa54600bcd657749a006d4b, keycount=150, bloomtype=ROW, size=12.7 K, encoding=NONE, compression=NONE, seqNum=458, earliestPutTs=1680922589157 2023-04-08 02:56:31,433 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting c0319156bd0542f08054e1121fa3882b, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=832, earliestPutTs=1680922590344 2023-04-08 02:56:31,433 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 327448e0c7fc43528e927a2b21699d0a, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=852, earliestPutTs=1680922590373 2023-04-08 02:56:31,436 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/2bbb486a8a724dfb83a301a9a0790c56 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/2bbb486a8a724dfb83a301a9a0790c56 2023-04-08 02:56:31,449 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 337c881227bdfa873bb568857a6d7e4b#C#compaction#663 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:31,452 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 337c881227bdfa873bb568857a6d7e4b/A of 337c881227bdfa873bb568857a6d7e4b into 2bbb486a8a724dfb83a301a9a0790c56(size=26.4 K), total size for store is 26.4 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:31,452 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:31,452 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., storeName=337c881227bdfa873bb568857a6d7e4b/A, priority=13, startTime=1680922591399; duration=0sec 2023-04-08 02:56:31,452 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:31,475 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:31,475 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:31,476 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:31,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:31,477 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:31,477 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:31,477 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:31,477 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:31,478 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:31,478 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,478 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:31,479 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:31,479 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,479 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:31,479 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:31,480 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,480 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:31,480 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:31,480 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:31,481 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:31,481 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:31,481 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:31,481 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:31,481 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:31,481 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:31,481 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:31,481 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:31,481 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:31,481 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:31,481 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:31,482 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-08 02:56:31,482 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:31,482 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:31,482 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:31,482 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:31,482 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:31,482 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:31,489 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:31,489 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:31,496 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408d7c200d87f3e46f7ad57cb0abf81aab7 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408d7c200d87f3e46f7ad57cb0abf81aab7 2023-04-08 02:56:31,530 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:31,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 188 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922651528, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,530 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:31,530 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 138 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46618 deadline: 1680922651529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,531 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:31,531 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 178 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46656 deadline: 1680922651529, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,579 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:31,579 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 337 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46582 deadline: 1680922651578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,580 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:31,581 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 335 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680922651578, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:31,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 190 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922651631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,632 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:31,632 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 140 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46618 deadline: 1680922651631, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,633 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:31,633 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 180 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46656 deadline: 1680922651632, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,835 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:31,835 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 192 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922651833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:31,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 142 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46618 deadline: 1680922651833, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,836 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:31,836 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 182 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46656 deadline: 1680922651834, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,862 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/81743c2d325a418abfb9306be9485e61 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/81743c2d325a418abfb9306be9485e61 2023-04-08 02:56:31,866 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 3 (all) file(s) in 337c881227bdfa873bb568857a6d7e4b/C of 337c881227bdfa873bb568857a6d7e4b into 81743c2d325a418abfb9306be9485e61(size=12.8 K), total size for store is 12.8 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:31,866 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:31,866 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., storeName=337c881227bdfa873bb568857a6d7e4b/C, priority=13, startTime=1680922591399; duration=0sec 2023-04-08 02:56:31,866 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:31,902 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=869, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/544beee08dfc4e37ad66220eafdccce8 2023-04-08 02:56:31,910 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=869 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/230dd3a209c14946874c18137ad17815 2023-04-08 02:56:31,918 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=869 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/1dd126b1990a435187d49bf5d0f953e4 2023-04-08 02:56:31,921 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/544beee08dfc4e37ad66220eafdccce8 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/544beee08dfc4e37ad66220eafdccce8 2023-04-08 02:56:31,924 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/544beee08dfc4e37ad66220eafdccce8, entries=150, sequenceid=869, filesize=25.4 K 2023-04-08 02:56:31,925 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/230dd3a209c14946874c18137ad17815 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/230dd3a209c14946874c18137ad17815 2023-04-08 02:56:31,928 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/230dd3a209c14946874c18137ad17815, entries=150, sequenceid=869, filesize=11.9 K 2023-04-08 02:56:31,928 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/1dd126b1990a435187d49bf5d0f953e4 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/1dd126b1990a435187d49bf5d0f953e4 2023-04-08 02:56:31,931 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/1dd126b1990a435187d49bf5d0f953e4, entries=150, sequenceid=869, filesize=11.9 K 2023-04-08 02:56:31,931 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 337c881227bdfa873bb568857a6d7e4b in 450ms, sequenceid=869, compaction requested=false 2023-04-08 02:56:31,932 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:31,932 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:31,932 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:31,932 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:31,932 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,932 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:31,932 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:31,933 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,933 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:31,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:31,934 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:31,934 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:31,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:31,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:31,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,935 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:31,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:31,936 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@8b94e77[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:31,936 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:31,936 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,938 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,938 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,938 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:31,938 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:31,938 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:31,938 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,938 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:31,939 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:31,939 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:31,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:31,939 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:31,940 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:31,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:31,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:31,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:31,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:31,941 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:31,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:31,943 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:31,943 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:31,943 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:31,943 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:31,943 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:31,943 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:31,957 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,957 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:31,957 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,957 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,958 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:31,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:31,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:31,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,959 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:31,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:31,960 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:31,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:31,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:31,963 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:31,963 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:31,963 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:31,963 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:31,963 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:31,963 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:31,963 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:31,963 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:31,963 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:31,963 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:31,963 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,963 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:31,963 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:31,964 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:31,963 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:31,964 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:31,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:31,964 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:32,064 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:32,064 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:32,066 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:32,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:32,067 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:32,068 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:32,068 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:32,068 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:32,068 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:32,068 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:32,068 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:32,070 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:32,070 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:32,070 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:32,071 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,071 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,071 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:32,072 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:32,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:32,072 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:32,072 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:32,072 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:32,072 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:32,072 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:32,072 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:32,072 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:32,073 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:32,073 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:32,073 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:32,073 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:32,073 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=93.93 KB heapSize=279.66 KB 2023-04-08 02:56:32,073 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:32,073 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:32,073 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:32,073 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:32,073 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:32,073 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:32,083 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:32,083 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:32,096 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408058a16854272472fa6ba21fd0a44a25d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408058a16854272472fa6ba21fd0a44a25d 2023-04-08 02:56:32,103 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:32,103 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 343 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680922652102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,103 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:32,104 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46582 deadline: 1680922652102, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,109 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=887, memsize=31.3 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/6e04514b4b1345358b906cd8f377e834 2023-04-08 02:56:32,139 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:32,139 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 184 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46656 deadline: 1680922652137, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:32,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 144 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46618 deadline: 1680922652138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,140 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:32,140 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 194 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922652138, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,206 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:32,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 345 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680922652204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,206 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:32,206 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46582 deadline: 1680922652204, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,409 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:32,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 347 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680922652407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,410 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:32,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46582 deadline: 1680922652408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,524 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=887 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/75c113a2e6e14be69a12eb8ade7dc675 2023-04-08 02:56:32,641 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:32,641 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 186 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46656 deadline: 1680922652640, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,644 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:32,644 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 146 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46618 deadline: 1680922652643, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,647 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:32,647 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 196 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922652645, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,714 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:32,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 349 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680922652712, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,715 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:32,715 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 351 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46582 deadline: 1680922652713, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,934 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=31.31 KB at sequenceid=887 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/a7731c79a7334ee49a59f82614d35486 2023-04-08 02:56:32,938 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/6e04514b4b1345358b906cd8f377e834 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/6e04514b4b1345358b906cd8f377e834 2023-04-08 02:56:32,941 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/6e04514b4b1345358b906cd8f377e834, entries=150, sequenceid=887, filesize=25.4 K 2023-04-08 02:56:32,942 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/75c113a2e6e14be69a12eb8ade7dc675 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/75c113a2e6e14be69a12eb8ade7dc675 2023-04-08 02:56:32,947 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/75c113a2e6e14be69a12eb8ade7dc675, entries=150, sequenceid=887, filesize=11.9 K 2023-04-08 02:56:32,948 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/a7731c79a7334ee49a59f82614d35486 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/a7731c79a7334ee49a59f82614d35486 2023-04-08 02:56:32,952 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/a7731c79a7334ee49a59f82614d35486, entries=150, sequenceid=887, filesize=11.9 K 2023-04-08 02:56:32,952 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~93.93 KB/96180, heapSize ~279.61 KB/286320, currentSize=80.51 KB/82440 for 337c881227bdfa873bb568857a6d7e4b in 879ms, sequenceid=887, compaction requested=true 2023-04-08 02:56:32,953 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:32,953 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:32,953 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:32,953 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:32,953 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,953 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:32,953 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:32,954 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:32,955 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:32,955 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:32,955 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:32,955 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,956 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:32,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:32,956 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@93c5166[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:32,956 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:32,957 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:32,958 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:32,958 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:32,958 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:32,958 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:32,958 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:32,958 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,958 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:32,958 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:32,959 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:32,959 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,959 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:32,959 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:32,959 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:32,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:32,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:32,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:32,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:32,962 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:32,962 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:32,962 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:32,962 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:32,963 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:32,963 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:32,966 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:32,966 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:32,966 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:32,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:32,966 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:32,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:32,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:32,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:32,967 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:32,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:32,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:32,968 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,968 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:32,969 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:32,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:32,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,971 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,977 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:32,977 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:32,977 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:32,977 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:32,977 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:32,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:32,977 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:32,977 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:32,977 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:32,977 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:32,977 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:32,977 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:32,977 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:32,978 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:32,978 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:32,977 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:33,078 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:33,078 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:33,079 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:33,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:33,081 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:33,081 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:33,081 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:33,081 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:33,081 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:33,081 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,081 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:33,082 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:33,082 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:33,082 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:33,082 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:33,083 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,083 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:33,083 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:33,083 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:33,083 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:33,083 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:33,083 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:33,084 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:33,084 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:33,084 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:33,084 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:33,084 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:33,084 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:33,084 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:33,084 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-08 02:56:33,084 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:33,084 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:33,084 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:33,084 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:33,084 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:33,084 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:33,097 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408f6c00a4d35514f7f9cc6b7e8fdc0c472 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408f6c00a4d35514f7f9cc6b7e8fdc0c472 2023-04-08 02:56:33,102 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=902, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/18e7df7da9494063a0d30dee4f9e185f 2023-04-08 02:56:33,140 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=902 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/1758a206e2ae45bba8b21c06921a12e6 2023-04-08 02:56:33,221 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:33,221 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:33,244 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:33,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 358 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680922653240, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,244 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:33,244 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46582 deadline: 1680922653242, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,346 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:33,346 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 360 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680922653345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,347 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:33,347 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46582 deadline: 1680922653345, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,549 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:33,549 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46582 deadline: 1680922653548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,550 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:33,550 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 362 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680922653548, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,557 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=902 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/42172418da164ca3a1bad08b192f4e45 2023-04-08 02:56:33,561 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/18e7df7da9494063a0d30dee4f9e185f as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/18e7df7da9494063a0d30dee4f9e185f 2023-04-08 02:56:33,568 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/18e7df7da9494063a0d30dee4f9e185f, entries=150, sequenceid=902, filesize=25.4 K 2023-04-08 02:56:33,570 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/1758a206e2ae45bba8b21c06921a12e6 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/1758a206e2ae45bba8b21c06921a12e6 2023-04-08 02:56:33,574 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/1758a206e2ae45bba8b21c06921a12e6, entries=150, sequenceid=902, filesize=11.9 K 2023-04-08 02:56:33,574 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/42172418da164ca3a1bad08b192f4e45 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/42172418da164ca3a1bad08b192f4e45 2023-04-08 02:56:33,577 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/42172418da164ca3a1bad08b192f4e45, entries=150, sequenceid=902, filesize=11.9 K 2023-04-08 02:56:33,579 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=93.93 KB/96180 for 337c881227bdfa873bb568857a6d7e4b in 495ms, sequenceid=902, compaction requested=true 2023-04-08 02:56:33,579 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:33,579 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:33,579 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:33,579 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:33,579 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,579 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:33,579 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:33,581 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,581 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:33,581 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:33,582 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,582 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:33,582 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:33,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:33,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,583 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:33,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:33,584 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@29905c9e[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:33,584 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:33,584 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,585 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,585 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,585 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,585 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:33,585 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:33,585 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:33,585 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,585 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:33,593 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:33,593 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:33,593 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:33,593 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:33,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:33,593 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:33,594 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:33,595 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,596 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:33,596 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:33,596 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:33,596 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:33,596 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:33,596 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:33,596 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:33,598 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,598 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:33,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:33,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:33,598 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,598 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,598 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:33,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:33,598 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,599 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:33,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:33,599 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:33,600 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,600 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:33,601 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,602 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,602 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:33,602 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:33,602 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:33,602 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,603 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,603 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,602 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:33,603 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:33,602 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:33,603 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:33,603 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:33,603 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:33,603 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:33,603 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:33,603 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:33,603 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:33,652 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:33,652 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-08 02:56:33,652 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:33,652 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:33,653 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:33,653 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:33,653 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:33,653 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:33,662 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304080acec8c50cba461fa1dc12b0841600d8 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304080acec8c50cba461fa1dc12b0841600d8 2023-04-08 02:56:33,666 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=920, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/42842c03d3bd4d928e48b5ff4e165ec2 2023-04-08 02:56:33,670 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:33,670 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46656 deadline: 1680922653667, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,672 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:33,672 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 202 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922653670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,673 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:33,673 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 152 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46618 deadline: 1680922653670, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,674 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=920 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/3f345fd87d5b49fc9e84d99853bbd643 2023-04-08 02:56:33,703 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:33,704 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:33,705 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:33,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:33,706 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:33,706 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:33,706 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:33,706 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:33,706 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:33,706 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,706 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:33,708 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:33,708 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,708 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:33,708 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:33,708 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,709 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:33,709 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:33,709 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:33,709 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:33,709 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:33,709 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:33,709 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:33,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:33,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:33,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:33,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:33,710 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:33,710 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:33,710 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:33,710 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:33,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:33,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:33,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:33,710 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:33,712 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:33,712 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:33,712 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,713 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:33,713 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:33,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:33,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:33,713 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,714 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:33,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:33,714 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@28f13be6[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:33,714 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:33,714 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,716 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,716 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,716 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,716 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:33,716 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:33,716 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:33,716 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,716 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:33,717 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,717 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:33,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:33,717 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:33,717 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:33,717 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:33,718 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:33,719 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:33,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,720 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:33,720 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:33,720 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:33,720 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:33,720 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:33,721 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:33,721 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:33,722 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,723 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,723 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:33,723 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:33,723 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:33,723 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:33,724 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:33,724 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:33,725 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:33,727 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,727 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:33,727 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:33,727 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:33,727 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:33,727 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:33,728 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,728 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:33,728 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:33,727 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:33,728 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,728 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,728 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:33,728 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:33,728 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:33,728 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:33,728 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:33,728 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:33,772 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:33,772 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46656 deadline: 1680922653771, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:33,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 154 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46618 deadline: 1680922653773, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,775 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:33,775 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 204 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922653774, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,828 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:33,829 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:33,830 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:33,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:33,831 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:33,831 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:33,832 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:33,832 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:33,832 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:33,832 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,832 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:33,833 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:33,833 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:33,834 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,834 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,834 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,834 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:33,834 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:33,834 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:33,835 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:33,835 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:33,835 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:33,835 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:33,835 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:33,835 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:33,835 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:33,835 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:33,835 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:33,835 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:33,835 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:33,835 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:33,835 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:33,835 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,835 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:33,835 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:33,837 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,837 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:33,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:33,837 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,837 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:33,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:33,837 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,838 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:33,838 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:33,839 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6a791279[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:33,839 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:33,839 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,840 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,840 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,840 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,840 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:33,840 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:33,840 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:33,841 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,841 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:33,843 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,843 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:33,843 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:33,843 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:33,843 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:33,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:33,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:33,844 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:33,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,845 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:33,845 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:33,845 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:33,845 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:33,845 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:33,845 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:33,845 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:33,847 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,847 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,847 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:33,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:33,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:33,847 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:33,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:33,847 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:33,847 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:33,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:33,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:33,848 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,848 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:33,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:33,849 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,853 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:33,853 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 364 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680922653851, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,854 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:33,854 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46582 deadline: 1680922653853, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,854 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:33,854 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:33,854 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:33,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:33,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:33,854 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:33,854 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:33,854 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:33,854 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:33,854 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,854 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:33,854 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:33,854 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:33,854 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,854 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:33,855 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,855 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:33,855 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,955 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:33,956 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:33,957 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:33,958 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:33,959 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:33,959 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:33,959 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:33,959 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:33,959 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:33,959 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,959 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:33,960 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:33,960 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,960 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:33,960 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:33,961 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,961 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:33,961 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:33,961 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:33,962 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:33,962 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:33,962 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:33,962 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:33,962 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:33,962 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:33,962 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:33,963 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:33,963 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:33,963 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:33,963 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:33,963 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:33,963 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:33,963 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:33,963 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:33,963 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,963 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:33,963 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:33,964 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:33,964 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,964 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:33,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:33,965 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,965 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:33,965 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:33,966 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:33,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,967 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:33,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:33,967 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@125d931[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:33,967 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:33,968 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,970 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,970 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,970 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,970 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:33,970 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:33,970 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:33,970 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,970 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:33,972 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,972 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:33,972 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:33,972 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:33,972 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:33,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:33,973 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:33,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,974 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:33,974 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 197 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46656 deadline: 1680922653973, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:33,974 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,975 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:33,975 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:33,975 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:33,975 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:33,975 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:33,975 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:33,975 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:33,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:33,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 156 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46618 deadline: 1680922653976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,977 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:33,977 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 206 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922653976, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,978 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,978 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:33,978 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,978 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:33,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:33,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,978 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:33,979 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:33,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:33,979 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,979 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:33,980 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:33,981 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:33,981 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,983 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,983 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:33,983 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:33,983 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:33,983 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,983 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:33,984 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:33,984 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:33,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:33,984 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:33,984 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:33,984 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:33,984 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:33,984 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:33,984 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:33,985 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:33,985 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:33,985 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:34,082 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=920 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/265a46bb9fca46a8af3c931db3922279 2023-04-08 02:56:34,085 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:34,085 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:34,087 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/42842c03d3bd4d928e48b5ff4e165ec2 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/42842c03d3bd4d928e48b5ff4e165ec2 2023-04-08 02:56:34,088 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:34,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:34,090 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:34,090 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:34,090 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:34,090 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:34,090 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:34,091 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:34,091 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:34,092 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/42842c03d3bd4d928e48b5ff4e165ec2, entries=200, sequenceid=920, filesize=32.3 K 2023-04-08 02:56:34,093 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/3f345fd87d5b49fc9e84d99853bbd643 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/3f345fd87d5b49fc9e84d99853bbd643 2023-04-08 02:56:34,093 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:34,093 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:34,093 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:34,094 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,094 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:34,094 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,094 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:34,094 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:34,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:34,095 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:34,095 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:34,096 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:34,096 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:34,096 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:34,096 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:34,096 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:34,097 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:34,097 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:34,097 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:34,097 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:34,097 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:34,097 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:34,098 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:34,098 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:34,098 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,098 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:34,098 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:34,099 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/3f345fd87d5b49fc9e84d99853bbd643, entries=150, sequenceid=920, filesize=11.9 K 2023-04-08 02:56:34,099 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,100 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:34,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:34,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:34,100 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/265a46bb9fca46a8af3c931db3922279 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/265a46bb9fca46a8af3c931db3922279 2023-04-08 02:56:34,100 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:34,100 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:34,100 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:34,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:34,101 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:34,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,102 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:34,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:34,102 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2afa0127[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:34,103 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:34,103 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:34,104 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:34,104 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:34,104 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:34,104 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:34,104 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:34,104 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,104 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:34,104 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:34,106 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:34,106 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:34,106 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:34,106 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:34,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:34,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:34,106 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:34,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:34,107 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/265a46bb9fca46a8af3c931db3922279, entries=150, sequenceid=920, filesize=11.9 K 2023-04-08 02:56:34,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,107 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:34,107 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=80.51 KB/82440 for 337c881227bdfa873bb568857a6d7e4b in 455ms, sequenceid=920, compaction requested=true 2023-04-08 02:56:34,108 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:34,108 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:34,108 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 02:56:34,108 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 02:56:34,108 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:34,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:34,108 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:34,108 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,109 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:34,109 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:34,109 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:34,109 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:34,109 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:34,109 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:34,109 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:34,110 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 138258 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 02:56:34,110 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 337c881227bdfa873bb568857a6d7e4b/A is initiating minor compaction (all files) 2023-04-08 02:56:34,110 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61918 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 02:56:34,110 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 337c881227bdfa873bb568857a6d7e4b/A in TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:34,110 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 337c881227bdfa873bb568857a6d7e4b/B is initiating minor compaction (all files) 2023-04-08 02:56:34,110 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/2bbb486a8a724dfb83a301a9a0790c56, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/544beee08dfc4e37ad66220eafdccce8, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/6e04514b4b1345358b906cd8f377e834, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/18e7df7da9494063a0d30dee4f9e185f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/42842c03d3bd4d928e48b5ff4e165ec2] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp, totalSize=135.0 K 2023-04-08 02:56:34,110 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 337c881227bdfa873bb568857a6d7e4b/B in TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:34,111 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/c7f8ce256513403a9958bf3fae4c413e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/230dd3a209c14946874c18137ad17815, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/75c113a2e6e14be69a12eb8ade7dc675, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/1758a206e2ae45bba8b21c06921a12e6, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/3f345fd87d5b49fc9e84d99853bbd643] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp, totalSize=60.5 K 2023-04-08 02:56:34,111 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 2bbb486a8a724dfb83a301a9a0790c56, keycount=150, bloomtype=ROW, size=26.4 K, encoding=NONE, compression=NONE, seqNum=852, earliestPutTs=1680922590373 2023-04-08 02:56:34,111 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting c7f8ce256513403a9958bf3fae4c413e, keycount=150, bloomtype=ROW, size=12.9 K, encoding=NONE, compression=NONE, seqNum=852, earliestPutTs=1680922590373 2023-04-08 02:56:34,111 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 544beee08dfc4e37ad66220eafdccce8, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=869, earliestPutTs=1680922590954 2023-04-08 02:56:34,111 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 230dd3a209c14946874c18137ad17815, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=869, earliestPutTs=1680922590954 2023-04-08 02:56:34,111 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 6e04514b4b1345358b906cd8f377e834, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=887, earliestPutTs=1680922591506 2023-04-08 02:56:34,112 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 75c113a2e6e14be69a12eb8ade7dc675, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=887, earliestPutTs=1680922591506 2023-04-08 02:56:34,112 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 18e7df7da9494063a0d30dee4f9e185f, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=902, earliestPutTs=1680922592089 2023-04-08 02:56:34,112 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 1758a206e2ae45bba8b21c06921a12e6, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=902, earliestPutTs=1680922592089 2023-04-08 02:56:34,112 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:34,112 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 3f345fd87d5b49fc9e84d99853bbd643, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=920, earliestPutTs=1680922593239 2023-04-08 02:56:34,112 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 42842c03d3bd4d928e48b5ff4e165ec2, keycount=200, bloomtype=ROW, size=32.3 K, encoding=NONE, compression=NONE, seqNum=920, earliestPutTs=1680922593230 2023-04-08 02:56:34,112 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:34,112 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:34,112 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:34,113 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:34,113 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:34,114 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,114 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:34,115 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:34,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:34,116 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,116 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:34,117 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:34,118 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,119 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:34,119 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:34,119 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:34,119 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:34,119 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,119 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:34,119 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:34,119 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:34,120 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:34,119 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:34,120 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,120 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:34,120 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:34,120 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:34,120 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:34,120 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:34,120 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:34,120 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:34,145 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 337c881227bdfa873bb568857a6d7e4b#B#compaction#677 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:34,154 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 337c881227bdfa873bb568857a6d7e4b#A#compaction#676 average throughput is 0.90 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:34,172 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/454c6df2ab7243f68776a1facda42272 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/454c6df2ab7243f68776a1facda42272 2023-04-08 02:56:34,176 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 337c881227bdfa873bb568857a6d7e4b/B of 337c881227bdfa873bb568857a6d7e4b into 454c6df2ab7243f68776a1facda42272(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:34,176 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:34,176 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., storeName=337c881227bdfa873bb568857a6d7e4b/B, priority=11, startTime=1680922594108; duration=0sec 2023-04-08 02:56:34,176 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:34,176 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 5 store files, 0 compacting, 5 eligible, 16 blocking 2023-04-08 02:56:34,178 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 5 files of size 61884 starting at candidate #0 after considering 6 permutations with 6 in ratio 2023-04-08 02:56:34,178 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 337c881227bdfa873bb568857a6d7e4b/C is initiating minor compaction (all files) 2023-04-08 02:56:34,178 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 337c881227bdfa873bb568857a6d7e4b/C in TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:34,178 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/81743c2d325a418abfb9306be9485e61, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/1dd126b1990a435187d49bf5d0f953e4, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/a7731c79a7334ee49a59f82614d35486, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/42172418da164ca3a1bad08b192f4e45, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/265a46bb9fca46a8af3c931db3922279] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp, totalSize=60.4 K 2023-04-08 02:56:34,178 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 81743c2d325a418abfb9306be9485e61, keycount=150, bloomtype=ROW, size=12.8 K, encoding=NONE, compression=NONE, seqNum=852, earliestPutTs=1680922590373 2023-04-08 02:56:34,179 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 1dd126b1990a435187d49bf5d0f953e4, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=869, earliestPutTs=1680922590954 2023-04-08 02:56:34,179 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting a7731c79a7334ee49a59f82614d35486, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=887, earliestPutTs=1680922591506 2023-04-08 02:56:34,180 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 42172418da164ca3a1bad08b192f4e45, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=902, earliestPutTs=1680922592089 2023-04-08 02:56:34,180 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 265a46bb9fca46a8af3c931db3922279, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=920, earliestPutTs=1680922593239 2023-04-08 02:56:34,202 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 337c881227bdfa873bb568857a6d7e4b#C#compaction#678 average throughput is 3.28 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:34,225 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:34,226 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:34,228 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:34,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:34,232 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:34,232 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:34,232 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:34,232 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:34,232 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:34,233 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:34,233 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:34,235 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:34,235 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,235 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:34,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:34,235 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,235 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:34,235 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:34,236 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:34,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:34,236 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:34,236 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:34,248 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:34,248 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:34,248 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:34,248 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:34,249 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:34,249 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:34,249 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:34,249 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:34,249 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=80.51 KB heapSize=239.81 KB 2023-04-08 02:56:34,249 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:34,249 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:34,249 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:34,249 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:34,250 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:34,250 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:34,279 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:34,279 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:34,306 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:34,306 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 212 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922654303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,307 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:34,307 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:34,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 163 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46618 deadline: 1680922654303, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,307 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 205 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46656 deadline: 1680922654305, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,357 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:34,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46582 deadline: 1680922654355, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,358 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:34,358 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 366 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680922654356, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,409 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:34,409 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 214 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922654407, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,410 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:34,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 207 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46656 deadline: 1680922654408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,411 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:34,411 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 165 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46618 deadline: 1680922654408, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,586 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/bfc7b7bfccc244708ecccbe839169248 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/bfc7b7bfccc244708ecccbe839169248 2023-04-08 02:56:34,590 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 337c881227bdfa873bb568857a6d7e4b/A of 337c881227bdfa873bb568857a6d7e4b into bfc7b7bfccc244708ecccbe839169248(size=26.6 K), total size for store is 26.6 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:34,590 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:34,590 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., storeName=337c881227bdfa873bb568857a6d7e4b/A, priority=11, startTime=1680922594108; duration=0sec 2023-04-08 02:56:34,590 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:34,611 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/51b9d3d645e348e7857332d51e5522bf as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/51b9d3d645e348e7857332d51e5522bf 2023-04-08 02:56:34,612 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:34,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 216 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922654610, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,612 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:34,612 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 209 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46656 deadline: 1680922654611, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,614 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 5 (all) file(s) in 337c881227bdfa873bb568857a6d7e4b/C of 337c881227bdfa873bb568857a6d7e4b into 51b9d3d645e348e7857332d51e5522bf(size=13.0 K), total size for store is 13.0 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:34,614 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:34,614 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., storeName=337c881227bdfa873bb568857a6d7e4b/C, priority=11, startTime=1680922594108; duration=0sec 2023-04-08 02:56:34,614 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:34,615 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:34,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 167 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46618 deadline: 1680922654613, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,674 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408cd8c78523b384bd4a14e10b54fe23d78 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408cd8c78523b384bd4a14e10b54fe23d78 2023-04-08 02:56:34,916 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:34,916 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 218 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922654915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,917 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:34,917 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 211 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46656 deadline: 1680922654915, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:34,919 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:34,919 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 169 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46618 deadline: 1680922654918, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,079 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=936, memsize=26.8 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/0cd178279194400f98433c18bb1420e8 2023-04-08 02:56:35,097 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=936 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/50aded6ea07042e79c3827d27790607e 2023-04-08 02:56:35,364 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:35,364 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46582 deadline: 1680922655363, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,367 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:35,368 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 368 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680922655367, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,422 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:35,422 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 213 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46656 deadline: 1680922655422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:35,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 220 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922655422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,423 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:35,423 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 171 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46618 deadline: 1680922655422, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,510 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=26.84 KB at sequenceid=936 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/6e5c88c093ca4f73a36d80b613aa3116 2023-04-08 02:56:35,513 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/0cd178279194400f98433c18bb1420e8 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/0cd178279194400f98433c18bb1420e8 2023-04-08 02:56:35,516 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/0cd178279194400f98433c18bb1420e8, entries=100, sequenceid=936, filesize=18.6 K 2023-04-08 02:56:35,517 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/50aded6ea07042e79c3827d27790607e as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/50aded6ea07042e79c3827d27790607e 2023-04-08 02:56:35,520 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/50aded6ea07042e79c3827d27790607e, entries=100, sequenceid=936, filesize=9.5 K 2023-04-08 02:56:35,521 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/6e5c88c093ca4f73a36d80b613aa3116 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/6e5c88c093ca4f73a36d80b613aa3116 2023-04-08 02:56:35,523 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/6e5c88c093ca4f73a36d80b613aa3116, entries=100, sequenceid=936, filesize=9.5 K 2023-04-08 02:56:35,524 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~80.51 KB/82440, heapSize ~239.77 KB/245520, currentSize=100.63 KB/103050 for 337c881227bdfa873bb568857a6d7e4b in 1275ms, sequenceid=936, compaction requested=false 2023-04-08 02:56:35,524 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:35,524 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:35,524 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:35,524 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:35,524 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,524 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:35,524 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:35,525 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,525 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:35,525 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:35,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:35,526 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:35,526 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:35,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:35,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:35,526 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:35,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,527 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:35,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:35,529 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@bf53195[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:35,529 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:35,529 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:35,530 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:35,530 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:35,531 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:35,531 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:35,531 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:35,531 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:35,531 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,531 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:35,532 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:35,532 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:35,532 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,532 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:35,532 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:35,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:35,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:35,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,533 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:35,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:35,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,534 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:35,534 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:35,534 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:35,534 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:35,534 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:35,534 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:35,534 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:35,536 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:35,536 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:35,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:35,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:35,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:35,536 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:35,536 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:35,536 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:35,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:35,537 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:35,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:35,537 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:35,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:35,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:35,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:35,537 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:35,538 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,538 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:35,539 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,540 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,540 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:35,540 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:35,540 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:35,540 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:35,540 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:35,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:35,540 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:35,541 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:35,540 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:35,541 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,541 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:35,541 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:35,541 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:35,541 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:35,541 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:35,541 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:35,541 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:35,641 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:35,642 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:35,644 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:35,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:35,645 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:35,645 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:35,645 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:35,645 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:35,645 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:35,645 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:35,645 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:35,651 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:35,651 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:35,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:35,651 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,651 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:35,652 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:35,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:35,652 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:35,652 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:35,652 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,652 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:35,652 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:35,652 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:35,652 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:35,653 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:35,653 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:35,653 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:35,653 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:35,653 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:35,653 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:35,654 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=100.63 KB heapSize=299.58 KB 2023-04-08 02:56:35,654 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:35,654 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:35,654 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:35,654 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:35,654 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:35,654 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:35,675 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408430dce0c677b4c919714656a4ef1b59c to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408430dce0c677b4c919714656a4ef1b59c 2023-04-08 02:56:35,683 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=956, memsize=33.5 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/d516fb12371945ab9580fb3bc866127b 2023-04-08 02:56:35,694 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=956 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/ecdaba59d8474dd79738fd5c0e0b3e23 2023-04-08 02:56:35,706 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=33.54 KB at sequenceid=956 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/25cb943541014d6d9b52e9419a73c178 2023-04-08 02:56:35,717 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/d516fb12371945ab9580fb3bc866127b as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/d516fb12371945ab9580fb3bc866127b 2023-04-08 02:56:35,722 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/d516fb12371945ab9580fb3bc866127b, entries=150, sequenceid=956, filesize=25.4 K 2023-04-08 02:56:35,723 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/ecdaba59d8474dd79738fd5c0e0b3e23 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/ecdaba59d8474dd79738fd5c0e0b3e23 2023-04-08 02:56:35,737 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/ecdaba59d8474dd79738fd5c0e0b3e23, entries=150, sequenceid=956, filesize=11.9 K 2023-04-08 02:56:35,738 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/25cb943541014d6d9b52e9419a73c178 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/25cb943541014d6d9b52e9419a73c178 2023-04-08 02:56:35,742 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/25cb943541014d6d9b52e9419a73c178, entries=150, sequenceid=956, filesize=11.9 K 2023-04-08 02:56:35,756 INFO [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2948): Finished flush of dataSize ~100.63 KB/103050, heapSize ~299.53 KB/306720, currentSize=0 B/0 for 337c881227bdfa873bb568857a6d7e4b in 102ms, sequenceid=956, compaction requested=true 2023-04-08 02:56:35,756 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:35,756 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:35,756 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:35,757 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:35,757 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,757 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:35,757 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:35,758 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:35,758 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:35,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:35,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:35,759 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:35,759 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:35,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:35,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:35,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,760 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:35,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:35,761 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@293d1048[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:35,761 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:35,761 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:35,762 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:35,762 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:35,762 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:35,762 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:35,762 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:35,762 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,762 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:35,762 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:35,764 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:35,765 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:35,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:35,765 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:35,765 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:35,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:35,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:35,765 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:35,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:35,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:35,766 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,767 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:35,767 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:35,767 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:35,767 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:35,767 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:35,767 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:35,767 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:35,769 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:35,770 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:35,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:35,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:35,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:35,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:35,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:35,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:35,770 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:35,770 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,770 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:35,770 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:35,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:35,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:35,771 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:35,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:35,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,772 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:35,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:35,773 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,774 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,781 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:35,781 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:35,781 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:35,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:35,781 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:35,782 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:35,782 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:35,782 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,782 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:35,782 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:35,782 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,782 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:35,782 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:35,782 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:35,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:35,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:35,783 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:35,783 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:35,882 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:35,882 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:35,884 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:35,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:35,896 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:35,896 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:35,896 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:35,896 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:35,896 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:35,897 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:35,897 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:35,898 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,898 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:35,898 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:35,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:35,898 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,898 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:35,898 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:35,899 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:35,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:35,899 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:35,899 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:35,901 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:35,901 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:35,901 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:35,901 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:35,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:35,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:35,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:35,902 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:35,902 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:35,902 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:35,902 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:35,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:35,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:35,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:35,902 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:35,904 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,904 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:35,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:35,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:35,904 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:35,904 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:35,904 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:35,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:35,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:35,905 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,906 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:35,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:35,906 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6feb80f[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:35,906 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:35,906 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:35,908 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:35,909 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:35,909 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:35,909 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:35,909 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,909 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:35,909 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:35,909 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:35,910 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:35,910 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:35,910 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:35,910 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:35,910 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:35,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:35,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:35,911 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:35,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:35,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:35,912 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,913 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:35,913 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:35,913 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:35,913 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:35,913 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:35,913 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:35,913 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:35,915 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:35,915 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:35,915 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:35,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:35,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:35,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:35,915 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:35,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:35,915 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:35,916 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:35,916 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:35,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:35,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:35,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:35,916 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:35,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,917 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:35,917 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:35,918 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,919 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,919 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:35,919 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:35,919 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:35,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:35,919 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:35,919 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:35,919 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:35,919 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:35,919 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:35,919 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:35,919 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:35,919 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:35,920 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:35,919 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:35,920 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:35,920 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:35,919 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:35,920 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:36,020 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:36,020 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:36,022 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:36,023 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:36,028 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:36,029 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:36,029 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:36,029 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:36,029 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:36,029 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,029 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,031 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:36,031 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,031 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:36,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:36,031 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,031 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:36,031 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,032 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:36,032 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,032 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:36,032 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:36,032 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:36,032 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:36,033 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:36,033 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:36,033 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:36,033 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:36,033 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:36,033 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:36,033 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:36,033 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:36,033 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:36,033 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:36,033 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,033 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:36,034 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:36,035 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,035 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:36,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:36,035 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,035 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:36,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:36,035 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:36,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,036 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:36,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:36,037 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@7c7762ce[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:36,037 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:36,037 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,038 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,038 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,038 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,038 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,038 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:36,038 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:36,038 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:36,038 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:36,053 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,053 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:36,053 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:36,053 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:36,054 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:36,054 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:36,055 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,056 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:36,056 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:36,056 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:36,056 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:36,056 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:36,056 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:36,056 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:36,059 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,059 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,059 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:36,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:36,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:36,059 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:36,059 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:36,059 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:36,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:36,060 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,060 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:36,061 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,062 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,063 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,063 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,063 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,063 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:36,063 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:36,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:36,063 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:36,063 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,063 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:36,063 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:36,063 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:36,063 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:36,063 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:36,063 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:36,063 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,064 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,064 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,064 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:36,064 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:36,164 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:36,164 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:36,166 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:36,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:36,167 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:36,167 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:36,168 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:36,168 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:36,168 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:36,168 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,168 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,169 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:36,169 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,169 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:36,169 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:36,170 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,170 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:36,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,170 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:36,170 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,170 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:36,171 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:36,171 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:36,171 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:36,171 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:36,171 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:36,171 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:36,171 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:36,171 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:36,171 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:36,172 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:36,172 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:36,172 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:36,172 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:36,172 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,172 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:36,172 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:36,173 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,173 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:36,173 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:36,174 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:36,174 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:36,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:36,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,174 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,175 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:36,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:36,175 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@5e2c9d4c[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:36,175 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:36,175 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,181 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,181 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,181 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,181 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:36,181 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:36,181 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:36,181 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,181 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:36,182 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,182 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:36,182 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:36,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:36,182 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:36,182 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:36,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:36,183 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:36,184 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,185 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:36,185 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:36,185 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:36,185 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:36,185 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:36,185 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:36,185 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:36,187 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,187 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:36,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:36,187 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,187 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:36,187 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,188 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:36,188 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:36,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:36,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,188 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,189 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,189 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:36,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,190 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,191 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:36,192 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:36,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:36,191 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,192 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:36,192 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,192 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,192 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,192 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,192 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,192 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:36,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:36,192 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:36,192 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:36,192 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:36,192 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:36,192 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:36,193 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:36,293 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:36,293 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:36,295 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:36,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:36,296 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:36,296 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:36,296 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:36,296 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:36,296 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:36,296 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,296 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,298 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:36,298 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,298 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:36,298 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:36,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,299 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,299 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:36,299 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:36,299 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,299 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:36,299 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:36,300 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:36,300 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:36,300 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:36,300 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:36,300 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:36,300 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:36,300 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:36,300 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:36,300 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:36,300 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:36,300 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:36,300 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:36,300 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,300 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:36,300 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:36,302 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,302 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:36,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:36,302 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,302 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:36,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:36,302 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:36,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,303 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:36,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:36,303 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1fd26991[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:36,303 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:36,304 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,305 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,305 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,305 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,305 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:36,305 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:36,305 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:36,305 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,305 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:36,306 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:36,306 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:36,306 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,306 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:36,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:36,306 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:36,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:36,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:36,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,307 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:36,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,308 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,309 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:36,309 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:36,309 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:36,309 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:36,309 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:36,309 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:36,309 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:36,311 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,311 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,311 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,311 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:36,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:36,311 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:36,312 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:36,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:36,312 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:36,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,312 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:36,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,313 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,313 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:36,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,314 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,315 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:36,315 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,315 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:36,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:36,315 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:36,315 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:36,315 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:36,315 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:36,316 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:36,315 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,316 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:36,316 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,315 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:36,316 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,316 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:36,316 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:36,316 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,316 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,316 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,416 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:36,416 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:36,418 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:36,419 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:36,420 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:36,420 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:36,420 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:36,420 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:36,420 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:36,420 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,420 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,422 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:36,422 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:36,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:36,422 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,422 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,422 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,422 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:36,422 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:36,423 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,423 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:36,423 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:36,423 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:36,423 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:36,423 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:36,423 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:36,423 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:36,424 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:36,424 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:36,424 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:36,424 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:36,424 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:36,424 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:36,424 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:36,424 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,424 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:36,424 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:36,425 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,425 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:36,425 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:36,426 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:36,426 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:36,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:36,426 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,427 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:36,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:36,427 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@6c9f84a4[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:36,427 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:36,427 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,429 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,429 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,429 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,429 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:36,429 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,429 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:36,429 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:36,429 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:36,431 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,431 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:36,431 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:36,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:36,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:36,431 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:36,431 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:36,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:36,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,432 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:36,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,433 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:36,433 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:36,433 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:36,433 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:36,433 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:36,433 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:36,433 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:36,435 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,435 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,435 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:36,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:36,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:36,435 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:36,435 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:36,435 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:36,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:36,436 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,437 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:36,437 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,438 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,447 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,447 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:36,447 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:36,447 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:36,447 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,447 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:36,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:36,447 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:36,447 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:36,447 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:36,447 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,447 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,447 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,447 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,447 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,447 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:36,447 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:36,448 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:36,448 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:36,450 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:36,450 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=46.96 KB heapSize=140.20 KB 2023-04-08 02:56:36,450 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:36,450 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:36,451 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:36,451 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:36,451 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:36,451 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:36,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922596466, max timestamp=1680922596469Num uniques 859; 2023-04-08 02:56:36,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-7] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:56:36,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922596466, max timestamp=1680922596469Num uniques 859; 2023-04-08 02:56:36,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=159, cellSize=7275, totalHeapSize=16776, min timestamp=1680922596466, max timestamp=1680922596469Num uniques 859; 2023-04-08 02:56:36,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-2] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:56:36,470 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-3] regionserver.CompactionPipeline(163): Suffix data size=39335, new segment data size=7275, suffix heap size=117064,new segment heap size=16776 　suffix off heap size=0, new segment off heap size=0, suffix cells count=859, new segment cells count=159 2023-04-08 02:56:36,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:36,504 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:36,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 187 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46618 deadline: 1680922656501, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,504 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 228 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46656 deadline: 1680922656502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,504 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:36,505 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922656502, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,548 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:36,548 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:36,550 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:36,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:36,551 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:36,551 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:36,551 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:36,551 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:36,551 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:36,552 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,552 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,554 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:36,554 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,554 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:36,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:36,554 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,554 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:36,554 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,555 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:36,555 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,555 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:36,555 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:36,555 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:36,555 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:36,555 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:36,555 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:36,556 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:36,556 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:36,556 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:36,556 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:36,556 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:36,556 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:36,556 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:36,556 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:36,556 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,556 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:36,556 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:36,557 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,557 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:36,557 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:36,558 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,558 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:36,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:36,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:36,558 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,559 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:36,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:36,559 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@2bb9eaeb[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:36,559 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:36,559 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,560 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,560 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,560 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,560 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:36,560 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:36,560 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:36,561 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,561 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:36,565 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,565 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:36,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:36,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:36,565 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:36,565 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:36,565 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:36,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:36,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,566 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:36,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,567 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:36,567 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:36,567 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:36,568 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:36,568 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:36,568 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:36,568 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:36,569 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,570 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,570 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:36,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:36,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:36,570 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,570 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:36,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:36,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:36,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,570 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,571 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,571 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:36,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,572 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,573 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,573 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,573 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,573 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,573 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:36,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:36,573 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:36,573 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,573 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,573 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,573 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:36,573 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:36,573 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:36,573 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:36,573 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:36,574 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:36,574 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:36,574 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:36,574 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:36,605 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:36,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 189 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46618 deadline: 1680922656605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,606 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:36,606 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 238 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922656605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,607 WARN [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:36,607 DEBUG [RpcServer.default.FPBQ.Fifo.handler=0,queue=0,port=35089] ipc.CallRunner(144): callId: 230 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46656 deadline: 1680922656605, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,674 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:36,674 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:36,676 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:36,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:36,677 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:36,677 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:36,677 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:36,677 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:36,677 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:36,678 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,678 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,679 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,679 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:36,679 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:36,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:36,679 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,679 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:36,679 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,679 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:36,680 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,680 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:36,680 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:36,680 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:36,680 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:36,680 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:36,680 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:36,680 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:36,680 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:36,680 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:36,681 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:36,681 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:36,681 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:36,681 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:36,681 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:36,681 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,681 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:36,681 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:36,683 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,683 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:36,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:36,683 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,683 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:36,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:36,683 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:36,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,684 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,685 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:36,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:36,685 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@11c5c42d[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:36,685 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:36,685 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,686 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,686 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,686 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,686 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:36,686 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:36,686 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:36,687 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,687 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:36,688 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:36,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:36,688 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:36,688 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:36,688 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:36,688 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:36,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:36,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,689 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:36,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,690 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:36,690 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:36,690 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:36,691 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:36,691 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:36,691 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:36,691 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:36,692 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,692 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:36,692 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,692 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:36,692 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:36,693 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:36,693 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,693 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:36,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,694 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,694 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:36,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,695 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,698 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:36,698 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:36,698 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:36,698 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:36,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:36,698 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:36,698 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:36,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:36,698 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,698 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,698 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,698 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:36,698 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:36,698 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:36,698 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,698 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:36,698 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,699 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,799 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:36,799 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:36,801 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:36,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:36,802 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:36,802 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:36,802 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:36,802 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:36,802 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:36,802 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,802 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,803 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,803 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:36,804 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:36,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:36,804 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,804 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:36,804 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,804 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:36,805 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,805 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:36,805 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:36,805 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:36,805 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:36,805 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:36,805 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:36,805 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:36,805 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:36,805 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:36,805 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:36,805 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:36,805 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:36,805 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:36,805 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:36,805 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,805 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:36,806 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:36,807 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:36,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:36,807 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,807 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,807 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:36,807 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:36,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:36,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:36,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 191 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46618 deadline: 1680922656807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,808 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:36,808 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 240 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922656807, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,808 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,809 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:36,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:36,809 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@1d57b965[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:36,809 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:36,809 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,810 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,810 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,810 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,810 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:36,810 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:36,810 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:36,810 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,811 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:36,811 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:36,811 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 232 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46656 deadline: 1680922656809, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,813 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,813 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:36,813 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:36,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:36,813 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:36,813 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:36,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:36,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:36,814 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:36,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,815 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,816 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:36,816 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:36,816 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:36,816 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:36,816 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:36,816 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:36,816 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:36,817 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,817 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,817 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:36,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:36,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:36,818 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:36,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:36,818 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,818 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:36,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:36,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,819 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:36,820 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,820 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,822 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,822 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:36,822 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:36,822 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:36,822 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:36,822 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,822 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,822 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,822 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,822 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:36,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:36,822 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:36,822 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,822 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:36,823 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:36,823 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:36,823 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:36,862 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e20230408683c3fe99bf74cdcb163a7609264053c to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408683c3fe99bf74cdcb163a7609264053c 2023-04-08 02:56:36,923 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:36,923 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:36,925 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(933): Client=jenkins//172.31.14.131 procedure request for: flush-table-proc 2023-04-08 02:56:36,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(143): Procedure TestAcidGuarantees was in running list but was completed. Accepting new attempt. 2023-04-08 02:56:36,926 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure.ProcedureCoordinator(165): Submitting procedure TestAcidGuarantees 2023-04-08 02:56:36,926 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(191): Starting procedure 'TestAcidGuarantees' 2023-04-08 02:56:36,926 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:36,926 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(199): Procedure 'TestAcidGuarantees' starting 'acquire' 2023-04-08 02:56:36,926 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(241): Starting procedure 'TestAcidGuarantees', kicking off acquire phase on members. 2023-04-08 02:56:36,927 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,927 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(92): Creating acquire znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,928 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:36,928 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(100): Watching for acquire node:/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,928 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:36,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:36,928 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,928 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(203): Waiting for all members to 'acquire' 2023-04-08 02:56:36,928 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(186): Found procedure znode: /hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,929 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(212): start proc data length is 4 2023-04-08 02:56:36,929 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(214): Found data for znode:/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,929 DEBUG [zk-event-processor-pool-0] flush.RegionServerFlushTableProcedureManager(153): Launching subprocedure to flush regions for TestAcidGuarantees 2023-04-08 02:56:36,929 WARN [zk-event-processor-pool-0] procedure.ProcedureMember(133): A completed old subproc TestAcidGuarantees is still present, removing 2023-04-08 02:56:36,929 DEBUG [zk-event-processor-pool-0] procedure.ProcedureMember(140): Submitting new Subprocedure:TestAcidGuarantees 2023-04-08 02:56:36,929 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(151): Starting subprocedure 'TestAcidGuarantees' with timeout 60000ms 2023-04-08 02:56:36,929 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(107): Scheduling process timer to run in: 60000 ms 2023-04-08 02:56:36,929 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(159): Subprocedure 'TestAcidGuarantees' starting 'acquire' stage 2023-04-08 02:56:36,930 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.FlushTableSubprocedure(113): Flush region tasks submitted for 1 regions 2023-04-08 02:56:36,930 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(242): Waiting for local region flush to finish. 2023-04-08 02:56:36,930 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(69): Starting region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:36,930 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(72): Flush region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. started... 2023-04-08 02:56:36,930 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] regionserver.HRegion(2404): NOT flushing TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. as already flushing 2023-04-08 02:56:36,930 DEBUG [rs(jenkins-hbase4.apache.org,35089,1680922417983)-flush-proc-pool-0] flush.FlushTableSubprocedure$RegionFlushTask(80): Closing region operation on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:36,930 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(253): Completed 1/1 local region flush tasks. 2023-04-08 02:56:36,930 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(255): Completed 1 local region flush tasks. 2023-04-08 02:56:36,930 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] flush.RegionServerFlushTableProcedureManager$FlushTableSubprocedurePool(287): cancelling 0 flush region tasks jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,930 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(161): Subprocedure 'TestAcidGuarantees' locally acquired 2023-04-08 02:56:36,930 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(242): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure (TestAcidGuarantees) in zk 2023-04-08 02:56:36,933 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,933 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(250): Watch for global barrier reached:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:36,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:36,933 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,933 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(166): Subprocedure 'TestAcidGuarantees' coordinator notified of 'acquire', waiting on 'reached' or 'abort' from coordinator 2023-04-08 02:56:36,933 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:36,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:36,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,934 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:36,934 DEBUG [zk-event-processor-pool-0] procedure.Procedure(291): member: 'jenkins-hbase4.apache.org,35089,1680922417983' joining acquired barrier for procedure 'TestAcidGuarantees' on coordinator 2023-04-08 02:56:36,935 DEBUG [zk-event-processor-pool-0] procedure.Procedure(300): Waiting on: java.util.concurrent.CountDownLatch@dfda4cf[Count = 0] remaining members to acquire global barrier 2023-04-08 02:56:36,935 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(207): Procedure 'TestAcidGuarantees' starting 'in-barrier' execution. 2023-04-08 02:56:36,935 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(116): Creating reached barrier zk node:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,936 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,936 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,936 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(128): Received reached global barrier:/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,936 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(180): Subprocedure 'TestAcidGuarantees' received 'reached' from coordinator. 2023-04-08 02:56:36,936 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(182): Subprocedure 'TestAcidGuarantees' locally completed 2023-04-08 02:56:36,936 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.ZKProcedureMemberRpcs(267): Marking procedure 'TestAcidGuarantees' completed for member 'jenkins-hbase4.apache.org,35089,1680922417983' in zk 2023-04-08 02:56:36,936 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,936 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(211): Waiting for all members to 'release' 2023-04-08 02:56:36,937 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,937 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(187): Subprocedure 'TestAcidGuarantees' has notified controller of completion 2023-04-08 02:56:36,937 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:36,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,937 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:36,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:36,937 DEBUG [member: 'jenkins-hbase4.apache.org,35089,1680922417983' subprocedure-pool-0] procedure.Subprocedure(212): Subprocedure 'TestAcidGuarantees' completed. 2023-04-08 02:56:36,938 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,939 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:36,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,940 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(218): Finished data from procedure 'TestAcidGuarantees' member 'jenkins-hbase4.apache.org,35089,1680922417983': 2023-04-08 02:56:36,941 DEBUG [zk-event-processor-pool-0] procedure.Procedure(321): Member: 'jenkins-hbase4.apache.org,35089,1680922417983' released barrier for procedure'TestAcidGuarantees', counting down latch. Waiting for 0 more 2023-04-08 02:56:36,941 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(216): Procedure 'TestAcidGuarantees' execution completed 2023-04-08 02:56:36,941 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(225): Running finish phase. 2023-04-08 02:56:36,941 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.Procedure(275): Finished coordinator procedure - removing self from list of running procedures 2023-04-08 02:56:36,941 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureCoordinator(162): Attempting to clean out zk node for op:TestAcidGuarantees 2023-04-08 02:56:36,941 INFO [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] procedure.ZKProcedureUtil(265): Clearing all znodes for procedure TestAcidGuaranteesincluding nodes /hbase/flush-table-proc/acquired /hbase/flush-table-proc/reached /hbase/flush-table-proc/abort 2023-04-08 02:56:36,942 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,942 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeCreated, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,942 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:36,942 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(77): Received created event:/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,942 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,943 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:36,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:36,943 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,943 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(316): Aborting procedure member for znode /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureCoordinator$1(194): Node created: /hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(225): Current zk system: 2023-04-08 02:56:36,944 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(227): |-/hbase/flush-table-proc 2023-04-08 02:56:36,944 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] zookeeper.ZKUtil(162): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on existing znode=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-abort 2023-04-08 02:56:36,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,945 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-acquired 2023-04-08 02:56:36,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |----TestAcidGuarantees 2023-04-08 02:56:36,946 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-------jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,950 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,950 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired 2023-04-08 02:56:36,950 DEBUG [(jenkins-hbase4.apache.org,46135,1680922416713)-proc-coordinator-pool-0] errorhandling.TimeoutExceptionInjector(87): Marking timer as complete - no error notifications will be received for this timer. 2023-04-08 02:56:36,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(180): Done waiting - exec procedure flush-table-proc for 'TestAcidGuarantees' 2023-04-08 02:56:36,950 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] flush.MasterFlushTableProcedureManager(182): Master flush table procedure is successful! 2023-04-08 02:56:36,950 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureUtil(244): |-reached 2023-04-08 02:56:36,950 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(104): Received procedure start children changed event: /hbase/flush-table-proc/acquired 2023-04-08 02:56:36,950 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(154): Looking for new procedures under znode:'/hbase/flush-table-proc/acquired' 2023-04-08 02:56:36,950 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/abort 2023-04-08 02:56:36,950 DEBUG [Thread-4172] client.HBaseAdmin(2690): Waiting a max of 300000 ms for procedure 'flush-table-proc : TestAcidGuarantees'' to complete. (max 20000 ms per retry) 2023-04-08 02:56:36,950 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/acquired/TestAcidGuarantees 2023-04-08 02:56:36,951 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:36,950 DEBUG [Thread-4172] client.HBaseAdmin(2698): (#1) Sleeping: 100ms while waiting for procedure completion. 2023-04-08 02:56:36,951 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,951 INFO [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs$1(107): Received procedure abort children changed event: /hbase/flush-table-proc/abort 2023-04-08 02:56:36,951 DEBUG [zk-event-processor-pool-0] procedure.ZKProcedureMemberRpcs(134): Checking for aborted procedures on node: '/hbase/flush-table-proc/abort' 2023-04-08 02:56:36,951 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/reached/TestAcidGuarantees 2023-04-08 02:56:36,951 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/flush-table-proc/abort/TestAcidGuarantees 2023-04-08 02:56:36,951 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/flush-table-proc/abort/TestAcidGuarantees because node does not exist (not an error) 2023-04-08 02:56:37,049 DEBUG [Thread-4179] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7993d99e to 127.0.0.1:64826 2023-04-08 02:56:37,049 DEBUG [Thread-4179] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:56:37,050 DEBUG [Thread-4181] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4fae3ed2 to 127.0.0.1:64826 2023-04-08 02:56:37,050 DEBUG [Thread-4181] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:56:37,050 DEBUG [Thread-4177] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x07d997c6 to 127.0.0.1:64826 2023-04-08 02:56:37,051 DEBUG [Thread-4177] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:56:37,051 DEBUG [Thread-4172] client.HBaseAdmin(2704): Getting current status of procedure from master... 2023-04-08 02:56:37,051 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1186): Checking to see if procedure from request:flush-table-proc is done 2023-04-08 02:56:37,053 DEBUG [Thread-4175] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x4f4eab6d to 127.0.0.1:64826 2023-04-08 02:56:37,053 DEBUG [Thread-4175] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:56:37,054 DEBUG [Thread-4173] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x66c25427 to 127.0.0.1:64826 2023-04-08 02:56:37,054 DEBUG [Thread-4173] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:56:37,111 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:37,111 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:37,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 193 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46618 deadline: 1680922657111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:37,111 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 242 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922657111, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:37,112 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:37,112 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 234 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46656 deadline: 1680922657112, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:37,267 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=971, memsize=15.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/f10b0966e3d146eaa746e6de3543500c 2023-04-08 02:56:37,376 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:37,376 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 372 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46582 deadline: 1680922657376, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:37,377 DEBUG [Thread-4166] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4134 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_2' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:56:37,381 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:37,381 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 370 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46610 deadline: 1680922657381, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:37,382 DEBUG [Thread-4168] client.RpcRetryingCallerImpl(129): Call exception, tries=6, retries=16, started=4141 ms ago, cancelled=false, msg=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) , details=row 'test_row_1' on table 'TestAcidGuarantees' at region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., hostname=jenkins-hbase4.apache.org,35089,1680922417983, seqNum=5, see https://s.apache.org/timeout, exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at sun.reflect.GeneratedConstructorAccessor27.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.instantiateException(RemoteWithExtrasException.java:97) at org.apache.hadoop.hbase.ipc.RemoteWithExtrasException.unwrapRemoteException(RemoteWithExtrasException.java:87) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.makeIOExceptionOfException(ProtobufUtil.java:276) at org.apache.hadoop.hbase.protobuf.ProtobufUtil.handleRemoteException(ProtobufUtil.java:261) at org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:126) at org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:104) at org.apache.hadoop.hbase.client.HTable.put(HTable.java:512) at org.apache.hadoop.hbase.AcidGuaranteesTestTool$AtomicityWriter.doAnAction(AcidGuaranteesTestTool.java:169) at org.apache.hadoop.hbase.MultithreadedTestUtil$RepeatingTestThread.doWork(MultithreadedTestUtil.java:148) at org.apache.hadoop.hbase.MultithreadedTestUtil$TestThread.run(MultithreadedTestUtil.java:122) Caused by: org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.RegionTooBusyException): org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.onCallFinished(AbstractRpcClient.java:382) at org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$100(AbstractRpcClient.java:88) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:416) at org.apache.hadoop.hbase.ipc.AbstractRpcClient$3.run(AbstractRpcClient.java:412) at org.apache.hadoop.hbase.ipc.Call.callComplete(Call.java:115) at org.apache.hadoop.hbase.ipc.Call.setException(Call.java:130) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.readResponse(NettyRpcDuplexHandler.java:162) at org.apache.hadoop.hbase.ipc.NettyRpcDuplexHandler.channelRead(NettyRpcDuplexHandler.java:192) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:346) at org.apache.hbase.thirdparty.io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:318) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:286) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) at org.apache.hbase.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) at org.apache.hbase.thirdparty.io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) at org.apache.hbase.thirdparty.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:724) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:650) at org.apache.hbase.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) at org.apache.hbase.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) at org.apache.hbase.thirdparty.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) at org.apache.hbase.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) at java.lang.Thread.run(Thread.java:750) 2023-04-08 02:56:37,613 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:37,613 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 195 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46618 deadline: 1680922657612, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:37,614 WARN [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:37,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=35089] ipc.CallRunner(144): callId: 244 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46594 deadline: 1680922657614, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:37,615 WARN [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(4965): Region is too busy due to exceeding memstore size limit. org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4963) at org.apache.hadoop.hbase.regionserver.HRegion.put(HRegion.java:3155) at org.apache.hadoop.hbase.regionserver.RSRpcServices.put(RSRpcServices.java:3006) at org.apache.hadoop.hbase.regionserver.RSRpcServices.mutate(RSRpcServices.java:2969) at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44994) at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:387) at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:132) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:369) at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:349) 2023-04-08 02:56:37,615 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] ipc.CallRunner(144): callId: 236 service: ClientService methodName: Mutate size: 4.7 K connection: 172.31.14.131:46656 deadline: 1680922657615, exception=org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 K, regionName=337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:37,675 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=971 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/bb3035d73e1949098981ccdbfc532882 2023-04-08 02:56:38,082 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=15.65 KB at sequenceid=971 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/9ae64b1bf8544423bad76cecf3bab60c 2023-04-08 02:56:38,085 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/f10b0966e3d146eaa746e6de3543500c as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/f10b0966e3d146eaa746e6de3543500c 2023-04-08 02:56:38,088 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/f10b0966e3d146eaa746e6de3543500c, entries=350, sequenceid=971, filesize=52.9 K 2023-04-08 02:56:38,088 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/bb3035d73e1949098981ccdbfc532882 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/bb3035d73e1949098981ccdbfc532882 2023-04-08 02:56:38,091 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/bb3035d73e1949098981ccdbfc532882, entries=150, sequenceid=971, filesize=11.9 K 2023-04-08 02:56:38,091 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/9ae64b1bf8544423bad76cecf3bab60c as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/9ae64b1bf8544423bad76cecf3bab60c 2023-04-08 02:56:38,093 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/9ae64b1bf8544423bad76cecf3bab60c, entries=150, sequenceid=971, filesize=11.9 K 2023-04-08 02:56:38,094 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~46.96 KB/48090, heapSize ~140.16 KB/143520, currentSize=134.18 KB/137400 for 337c881227bdfa873bb568857a6d7e4b in 1644ms, sequenceid=971, compaction requested=true 2023-04-08 02:56:38,094 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:38,094 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:38,094 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:38,094 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:56:38,094 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:56:38,094 DEBUG [MemStoreFlusher.0] regionserver.CompactSplit(385): Small Compaction requested: system; Because: MemStoreFlusher.0; compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:38,095 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 126462 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:56:38,095 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47458 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:56:38,095 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 337c881227bdfa873bb568857a6d7e4b/A is initiating minor compaction (all files) 2023-04-08 02:56:38,095 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1912): 337c881227bdfa873bb568857a6d7e4b/B is initiating minor compaction (all files) 2023-04-08 02:56:38,095 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 337c881227bdfa873bb568857a6d7e4b/A in TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:38,095 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2259): Starting compaction of 337c881227bdfa873bb568857a6d7e4b/B in TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:38,095 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/bfc7b7bfccc244708ecccbe839169248, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/0cd178279194400f98433c18bb1420e8, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/d516fb12371945ab9580fb3bc866127b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/f10b0966e3d146eaa746e6de3543500c] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp, totalSize=123.5 K 2023-04-08 02:56:38,095 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/454c6df2ab7243f68776a1facda42272, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/50aded6ea07042e79c3827d27790607e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/ecdaba59d8474dd79738fd5c0e0b3e23, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/bb3035d73e1949098981ccdbfc532882] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp, totalSize=46.3 K 2023-04-08 02:56:38,096 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting bfc7b7bfccc244708ecccbe839169248, keycount=150, bloomtype=ROW, size=26.6 K, encoding=NONE, compression=NONE, seqNum=920, earliestPutTs=1680922593239 2023-04-08 02:56:38,096 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 454c6df2ab7243f68776a1facda42272, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=920, earliestPutTs=1680922593239 2023-04-08 02:56:38,096 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 0cd178279194400f98433c18bb1420e8, keycount=100, bloomtype=ROW, size=18.6 K, encoding=NONE, compression=NONE, seqNum=936, earliestPutTs=1680922593665 2023-04-08 02:56:38,096 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting 50aded6ea07042e79c3827d27790607e, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=936, earliestPutTs=1680922593665 2023-04-08 02:56:38,096 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting d516fb12371945ab9580fb3bc866127b, keycount=150, bloomtype=ROW, size=25.4 K, encoding=NONE, compression=NONE, seqNum=956, earliestPutTs=1680922594293 2023-04-08 02:56:38,096 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting ecdaba59d8474dd79738fd5c0e0b3e23, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=956, earliestPutTs=1680922594293 2023-04-08 02:56:38,096 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting f10b0966e3d146eaa746e6de3543500c, keycount=350, bloomtype=ROW, size=52.9 K, encoding=NONE, compression=NONE, seqNum=971, earliestPutTs=1680922596438 2023-04-08 02:56:38,096 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] compactions.Compactor(207): Compacting bb3035d73e1949098981ccdbfc532882, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=971, earliestPutTs=1680922596439 2023-04-08 02:56:38,102 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] throttle.PressureAwareThroughputController(145): 337c881227bdfa873bb568857a6d7e4b#B#compaction#689 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 1 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:38,103 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 337c881227bdfa873bb568857a6d7e4b#A#compaction#688 average throughput is 9.86 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:38,114 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/52ac88894a7743119903a24ecab900cc as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/52ac88894a7743119903a24ecab900cc 2023-04-08 02:56:38,117 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 337c881227bdfa873bb568857a6d7e4b/A of 337c881227bdfa873bb568857a6d7e4b into 52ac88894a7743119903a24ecab900cc(size=26.7 K), total size for store is 26.7 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:38,117 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:38,117 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., storeName=337c881227bdfa873bb568857a6d7e4b/A, priority=12, startTime=1680922598094; duration=0sec 2023-04-08 02:56:38,117 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=1), splitQueue=0 2023-04-08 02:56:38,117 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.SortedCompactionPolicy(75): Selecting compaction from 4 store files, 0 compacting, 4 eligible, 16 blocking 2023-04-08 02:56:38,118 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.ExploringCompactionPolicy(116): Exploring compaction algorithm has selected 4 files of size 47424 starting at candidate #0 after considering 3 permutations with 3 in ratio 2023-04-08 02:56:38,118 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1912): 337c881227bdfa873bb568857a6d7e4b/C is initiating minor compaction (all files) 2023-04-08 02:56:38,118 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2259): Starting compaction of 337c881227bdfa873bb568857a6d7e4b/C in TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:38,118 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1468): Starting compaction of [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/51b9d3d645e348e7857332d51e5522bf, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/6e5c88c093ca4f73a36d80b613aa3116, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/25cb943541014d6d9b52e9419a73c178, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/9ae64b1bf8544423bad76cecf3bab60c] into tmpdir=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp, totalSize=46.3 K 2023-04-08 02:56:38,119 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 51b9d3d645e348e7857332d51e5522bf, keycount=150, bloomtype=ROW, size=13.0 K, encoding=NONE, compression=NONE, seqNum=920, earliestPutTs=1680922593239 2023-04-08 02:56:38,119 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 6e5c88c093ca4f73a36d80b613aa3116, keycount=100, bloomtype=ROW, size=9.5 K, encoding=NONE, compression=NONE, seqNum=936, earliestPutTs=1680922593665 2023-04-08 02:56:38,119 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 25cb943541014d6d9b52e9419a73c178, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=956, earliestPutTs=1680922594293 2023-04-08 02:56:38,119 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] compactions.Compactor(207): Compacting 9ae64b1bf8544423bad76cecf3bab60c, keycount=150, bloomtype=ROW, size=11.9 K, encoding=NONE, compression=NONE, seqNum=971, earliestPutTs=1680922596439 2023-04-08 02:56:38,124 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] throttle.PressureAwareThroughputController(145): 337c881227bdfa873bb568857a6d7e4b#C#compaction#690 average throughput is 6.55 MB/second, slept 0 time(s) and total slept time is 0 ms. 0 active operations remaining, total limit is 50.00 MB/second 2023-04-08 02:56:38,129 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/9d0ab4a6fcdc40168cdf09021f6ff464 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/9d0ab4a6fcdc40168cdf09021f6ff464 2023-04-08 02:56:38,132 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 337c881227bdfa873bb568857a6d7e4b/C of 337c881227bdfa873bb568857a6d7e4b into 9d0ab4a6fcdc40168cdf09021f6ff464(size=13.1 K), total size for store is 13.1 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:38,132 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.HRegion(2289): Compaction status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:38,132 INFO [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., storeName=337c881227bdfa873bb568857a6d7e4b/C, priority=12, startTime=1680922598094; duration=0sec 2023-04-08 02:56:38,132 DEBUG [RS:0;jenkins-hbase4:35089-shortCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:38,509 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/22b139cb3a3b4bc69d051bd20fa86a30 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/22b139cb3a3b4bc69d051bd20fa86a30 2023-04-08 02:56:38,512 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HStore(1652): Completed compaction of 4 (all) file(s) in 337c881227bdfa873bb568857a6d7e4b/B of 337c881227bdfa873bb568857a6d7e4b into 22b139cb3a3b4bc69d051bd20fa86a30(size=13.2 K), total size for store is 13.2 K. This selection was in queue for 0sec, and took 0sec to execute. 2023-04-08 02:56:38,512 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.HRegion(2289): Compaction status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:38,512 INFO [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(627): Completed compaction region=TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b., storeName=337c881227bdfa873bb568857a6d7e4b/B, priority=12, startTime=1680922598094; duration=0sec 2023-04-08 02:56:38,512 DEBUG [RS:0;jenkins-hbase4:35089-longCompactions-0] regionserver.CompactSplit$CompactionRunner(659): Status compactionQueue=(longCompactions=0:shortCompactions=0), splitQueue=0 2023-04-08 02:56:38,618 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=35089] regionserver.HRegion(9158): Flush requested on 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:38,618 INFO [MemStoreFlusher.0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=140.89 KB heapSize=405.61 KB 2023-04-08 02:56:38,618 DEBUG [Thread-4164] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6af74c0e to 127.0.0.1:64826 2023-04-08 02:56:38,618 DEBUG [Thread-4164] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:56:38,619 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680922596480, max timestamp=1680922598617Num uniques 1018; 2023-04-08 02:56:38,619 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680922596480, max timestamp=1680922598617Num uniques 1018; 2023-04-08 02:56:38,619 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:38,619 DEBUG [Thread-4170] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1fbb5b46 to 127.0.0.1:64826 2023-04-08 02:56:38,619 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:38,619 DEBUG [Thread-4170] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:56:38,619 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:38,619 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-5] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-08 02:56:38,619 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-08 02:56:38,619 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:38,619 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=type=CellArrayImmutableSegment, empty=no, cellCount=168, cellSize=7688, totalHeapSize=17712, min timestamp=1680922596480, max timestamp=1680922598617Num uniques 1018; 2023-04-08 02:56:38,619 DEBUG [MemStoreFlusher.0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:38,619 DEBUG [RS_IN_MEMORY_COMPACTION-regionserver/jenkins-hbase4:0-9] regionserver.CompactionPipeline(163): Suffix data size=46618, new segment data size=7688, suffix heap size=133840,new segment heap size=17712 　suffix off heap size=0, new segment off heap size=0, suffix cells count=1018, new segment cells count=168 2023-04-08 02:56:38,619 DEBUG [MemStoreFlusher.0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=2, new segment=null 2023-04-08 02:56:38,623 DEBUG [Thread-4162] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1644730f to 127.0.0.1:64826 2023-04-08 02:56:38,623 DEBUG [Thread-4162] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:56:39,028 INFO [MemStoreFlusher.0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e202304082ceac21d52e643159fcda723f3201cfb to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304082ceac21d52e643159fcda723f3201cfb 2023-04-08 02:56:39,432 INFO [MemStoreFlusher.0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1013, memsize=11.2 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/986c155ef0d14dd6bfa957d02a4f9294 2023-04-08 02:56:39,732 DEBUG [regionserver/jenkins-hbase4:0.Chore.1] throttle.PressureAwareCompactionThroughputController(103): CompactionPressure is 0.0, tune throughput to 50.00 MB/second 2023-04-08 02:56:39,840 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=1013 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/e74e69ecc7114b0ba721c921ea86505c 2023-04-08 02:56:40,247 INFO [MemStoreFlusher.0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=11.18 KB at sequenceid=1013 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/02e7dade70de45508852d7fb8e5574dc 2023-04-08 02:56:40,250 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/986c155ef0d14dd6bfa957d02a4f9294 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/986c155ef0d14dd6bfa957d02a4f9294 2023-04-08 02:56:40,253 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/986c155ef0d14dd6bfa957d02a4f9294, entries=150, sequenceid=1013, filesize=25.4 K 2023-04-08 02:56:40,253 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/e74e69ecc7114b0ba721c921ea86505c as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/e74e69ecc7114b0ba721c921ea86505c 2023-04-08 02:56:40,256 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/e74e69ecc7114b0ba721c921ea86505c, entries=150, sequenceid=1013, filesize=11.9 K 2023-04-08 02:56:40,256 DEBUG [MemStoreFlusher.0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/02e7dade70de45508852d7fb8e5574dc as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/02e7dade70de45508852d7fb8e5574dc 2023-04-08 02:56:40,259 INFO [MemStoreFlusher.0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/02e7dade70de45508852d7fb8e5574dc, entries=150, sequenceid=1013, filesize=11.9 K 2023-04-08 02:56:40,259 INFO [MemStoreFlusher.0] regionserver.HRegion(2948): Finished flush of dataSize ~33.54 KB/34350, heapSize ~85.27 KB/87312, currentSize=6.71 KB/6870 for 337c881227bdfa873bb568857a6d7e4b in 1641ms, sequenceid=1013, compaction requested=false 2023-04-08 02:56:40,260 DEBUG [MemStoreFlusher.0] regionserver.HRegion(2446): Flush status journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:41,407 DEBUG [Thread-4166] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x1d759aab to 127.0.0.1:64826 2023-04-08 02:56:41,407 DEBUG [Thread-4168] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x39241fe3 to 127.0.0.1:64826 2023-04-08 02:56:41,408 DEBUG [Thread-4166] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:56:41,408 DEBUG [Thread-4168] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:56:41,408 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(392): Finished test. Writers: 2023-04-08 02:56:41,408 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 121 2023-04-08 02:56:41,408 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 129 2023-04-08 02:56:41,408 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 269 2023-04-08 02:56:41,408 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 267 2023-04-08 02:56:41,408 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(394): wrote 102 2023-04-08 02:56:41,408 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(396): Readers: 2023-04-08 02:56:41,408 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(398): read 4921 2023-04-08 02:56:41,408 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(398): read 5006 2023-04-08 02:56:41,408 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(398): read 4946 2023-04-08 02:56:41,408 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(398): read 4909 2023-04-08 02:56:41,408 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(398): read 5022 2023-04-08 02:56:41,408 INFO [Listener at localhost/39741] hbase.AcidGuaranteesTestTool(400): Scanners: 2023-04-08 02:56:41,408 INFO [Listener at localhost/39741] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-08 02:56:41,408 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x45878790 to 127.0.0.1:64826 2023-04-08 02:56:41,409 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:56:41,409 INFO [Listener at localhost/39741] client.HBaseAdmin$15(890): Started disable of TestAcidGuarantees 2023-04-08 02:56:41,409 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.HMaster$11(2418): Client=jenkins//172.31.14.131 disable TestAcidGuarantees 2023-04-08 02:56:41,410 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure2.ProcedureExecutor(1029): Stored pid=62, state=RUNNABLE:DISABLE_TABLE_PREPARE; DisableTableProcedure table=TestAcidGuarantees 2023-04-08 02:56:41,412 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-08 02:56:41,412 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680922601412"}]},"ts":"1680922601412"} 2023-04-08 02:56:41,413 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLING in hbase:meta 2023-04-08 02:56:41,414 INFO [PEWorker-5] procedure.DisableTableProcedure(293): Set TestAcidGuarantees to state=DISABLING 2023-04-08 02:56:41,415 INFO [PEWorker-5] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=337c881227bdfa873bb568857a6d7e4b, UNASSIGN}] 2023-04-08 02:56:41,416 INFO [PEWorker-1] procedure.MasterProcedureScheduler(727): Took xlock for pid=63, ppid=62, state=RUNNABLE:REGION_STATE_TRANSITION_CLOSE; TransitRegionStateProcedure table=TestAcidGuarantees, region=337c881227bdfa873bb568857a6d7e4b, UNASSIGN 2023-04-08 02:56:41,416 INFO [PEWorker-1] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=337c881227bdfa873bb568857a6d7e4b, regionState=CLOSING, regionLocation=jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:41,416 DEBUG [PEWorker-1] assignment.RegionStateStore(405): Put {"totalColumns":3,"row":"TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922601416"},{"qualifier":"sn","vlen":45,"tag":[],"timestamp":"1680922601416"},{"qualifier":"state","vlen":7,"tag":[],"timestamp":"1680922601416"}]},"ts":"1680922601416"} 2023-04-08 02:56:41,417 INFO [PEWorker-1] procedure2.ProcedureExecutor(1681): Initialized subprocedures=[{pid=64, ppid=63, state=RUNNABLE; CloseRegionProcedure 337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983}] 2023-04-08 02:56:41,513 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-08 02:56:41,569 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(109): Close 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:41,569 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 337c881227bdfa873bb568857a6d7e4b, disabling compactions & flushes 2023-04-08 02:56:41,569 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:41,569 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:41,569 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. after waiting 0 ms 2023-04-08 02:56:41,569 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:41,569 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 337c881227bdfa873bb568857a6d7e4b 3/3 column families, dataSize=20.13 KB heapSize=60.52 KB 2023-04-08 02:56:41,569 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=A 2023-04-08 02:56:41,569 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:41,569 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=B 2023-04-08 02:56:41,569 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:41,569 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactingMemStore(204): FLUSHING TO DISK 337c881227bdfa873bb568857a6d7e4b, store=C 2023-04-08 02:56:41,569 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.CompactionPipeline(136): Swapping pipeline suffix; before=1, new segment=null 2023-04-08 02:56:41,713 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-08 02:56:41,978 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HMobStore(257): Renaming flushed file from hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/.tmp/d41d8cd98f00b204e9800998ecf8427e2023040890e9cd149fcd4e7ca6274a55f3c211b8 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040890e9cd149fcd4e7ca6274a55f3c211b8 2023-04-08 02:56:42,014 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-08 02:56:42,383 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] mob.DefaultMobStoreFlusher(134): Mob store is flushed, sequenceid=1019, memsize=6.7 K, hasBloomFilter=true, into tmp file hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/78eeab7a694840d2988fc6d886ade9e9 2023-04-08 02:56:42,515 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-08 02:56:42,790 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1019 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/f473a2d6f0ed429e95fa9425f292b961 2023-04-08 02:56:43,198 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=6.71 KB at sequenceid=1019 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/7bfdffabe2c24952aa34b60bb10225fd 2023-04-08 02:56:43,201 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/A/78eeab7a694840d2988fc6d886ade9e9 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/78eeab7a694840d2988fc6d886ade9e9 2023-04-08 02:56:43,204 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/78eeab7a694840d2988fc6d886ade9e9, entries=150, sequenceid=1019, filesize=25.4 K 2023-04-08 02:56:43,204 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/B/f473a2d6f0ed429e95fa9425f292b961 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/f473a2d6f0ed429e95fa9425f292b961 2023-04-08 02:56:43,207 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/f473a2d6f0ed429e95fa9425f292b961, entries=150, sequenceid=1019, filesize=11.9 K 2023-04-08 02:56:43,207 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/.tmp/C/7bfdffabe2c24952aa34b60bb10225fd as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/7bfdffabe2c24952aa34b60bb10225fd 2023-04-08 02:56:43,210 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/7bfdffabe2c24952aa34b60bb10225fd, entries=150, sequenceid=1019, filesize=11.9 K 2023-04-08 02:56:43,211 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.13 KB/20610, heapSize ~60.47 KB/61920, currentSize=0 B/0 for 337c881227bdfa873bb568857a6d7e4b in 1642ms, sequenceid=1019, compaction requested=true 2023-04-08 02:56:43,236 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/d7a3cf0c2afc4af1b50007bc0f6ecef3, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/5cef7f52deb441e5b3530bd9736e475a, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/3b1a4df88f5d470c8a4f5cbed136b6a1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/c70d804639fa4d6a93344ae5668cc7c5, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/4bf7b447754f447ea4f4f97999956081, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/1ee939c85e3b4b4f8d4503a57d519a82, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/7dcb76f2ac7b47e3986eb18323a007d5, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/7869dde691b845dda59e07aca00dfb07, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/839e455c04cf4259affa82b7690c6c28, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/6d22543d880e4655bb7b91dc4c4eeb08, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/1681a8bb820d4a83ad933f5d26ab2fa3, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/5faacd544c4141098f59ab143c0bf3e0, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/7ae31d6bc02141edbbd2ad92db56764c, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/12e56920ca58499c84a41c069b51f1f4, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/6816d59a1f704876b767774761f45c26, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/f6a99ba0e8f446888fed4a9cd8b08c88, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/360a984c7d6e4db19c326cf4f62a1c9c, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/3af7315233b4426bbec6e82a4f53ffc0, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/10a6fdb7c2104dfc9054772005677ac1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/2624809fe2894b9fbb5f804f8e5f440d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/c0ba1ed5554f4cb187f24aad6fa752bd, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/96785c1e3ab042adbddddd50def28a14, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/47cceee4253d4eab93e0df0a2adee5b2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/2485e3c30ba047b2b60cacacc9d29b7d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/6cfa536c55594bdabb40010912698049, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/a54a66c1701f4d94b46554322126fa6b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/dd2591ebd6c44801a5566da8d1048c3b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/c4a9e0f392ba47b9b7bc3582907915c6, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/a146727b257547aead830f6b888f312b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/2bbb486a8a724dfb83a301a9a0790c56, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/544beee08dfc4e37ad66220eafdccce8, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/6e04514b4b1345358b906cd8f377e834, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/18e7df7da9494063a0d30dee4f9e185f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/42842c03d3bd4d928e48b5ff4e165ec2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/bfc7b7bfccc244708ecccbe839169248, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/0cd178279194400f98433c18bb1420e8, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/d516fb12371945ab9580fb3bc866127b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/f10b0966e3d146eaa746e6de3543500c] to archive 2023-04-08 02:56:43,237 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 02:56:43,238 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/d7a3cf0c2afc4af1b50007bc0f6ecef3 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/d7a3cf0c2afc4af1b50007bc0f6ecef3 2023-04-08 02:56:43,239 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/5cef7f52deb441e5b3530bd9736e475a to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/5cef7f52deb441e5b3530bd9736e475a 2023-04-08 02:56:43,240 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/3b1a4df88f5d470c8a4f5cbed136b6a1 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/3b1a4df88f5d470c8a4f5cbed136b6a1 2023-04-08 02:56:43,241 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/c70d804639fa4d6a93344ae5668cc7c5 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/c70d804639fa4d6a93344ae5668cc7c5 2023-04-08 02:56:43,241 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/4bf7b447754f447ea4f4f97999956081 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/4bf7b447754f447ea4f4f97999956081 2023-04-08 02:56:43,242 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/1ee939c85e3b4b4f8d4503a57d519a82 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/1ee939c85e3b4b4f8d4503a57d519a82 2023-04-08 02:56:43,243 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/7dcb76f2ac7b47e3986eb18323a007d5 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/7dcb76f2ac7b47e3986eb18323a007d5 2023-04-08 02:56:43,244 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/7869dde691b845dda59e07aca00dfb07 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/7869dde691b845dda59e07aca00dfb07 2023-04-08 02:56:43,245 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/839e455c04cf4259affa82b7690c6c28 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/839e455c04cf4259affa82b7690c6c28 2023-04-08 02:56:43,246 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/6d22543d880e4655bb7b91dc4c4eeb08 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/6d22543d880e4655bb7b91dc4c4eeb08 2023-04-08 02:56:43,246 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/1681a8bb820d4a83ad933f5d26ab2fa3 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/1681a8bb820d4a83ad933f5d26ab2fa3 2023-04-08 02:56:43,247 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/5faacd544c4141098f59ab143c0bf3e0 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/5faacd544c4141098f59ab143c0bf3e0 2023-04-08 02:56:43,248 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/7ae31d6bc02141edbbd2ad92db56764c to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/7ae31d6bc02141edbbd2ad92db56764c 2023-04-08 02:56:43,249 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/12e56920ca58499c84a41c069b51f1f4 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/12e56920ca58499c84a41c069b51f1f4 2023-04-08 02:56:43,250 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/6816d59a1f704876b767774761f45c26 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/6816d59a1f704876b767774761f45c26 2023-04-08 02:56:43,251 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/f6a99ba0e8f446888fed4a9cd8b08c88 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/f6a99ba0e8f446888fed4a9cd8b08c88 2023-04-08 02:56:43,252 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/360a984c7d6e4db19c326cf4f62a1c9c to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/360a984c7d6e4db19c326cf4f62a1c9c 2023-04-08 02:56:43,253 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/3af7315233b4426bbec6e82a4f53ffc0 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/3af7315233b4426bbec6e82a4f53ffc0 2023-04-08 02:56:43,253 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/10a6fdb7c2104dfc9054772005677ac1 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/10a6fdb7c2104dfc9054772005677ac1 2023-04-08 02:56:43,254 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/2624809fe2894b9fbb5f804f8e5f440d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/2624809fe2894b9fbb5f804f8e5f440d 2023-04-08 02:56:43,255 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/c0ba1ed5554f4cb187f24aad6fa752bd to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/c0ba1ed5554f4cb187f24aad6fa752bd 2023-04-08 02:56:43,256 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/96785c1e3ab042adbddddd50def28a14 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/96785c1e3ab042adbddddd50def28a14 2023-04-08 02:56:43,257 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/47cceee4253d4eab93e0df0a2adee5b2 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/47cceee4253d4eab93e0df0a2adee5b2 2023-04-08 02:56:43,258 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/2485e3c30ba047b2b60cacacc9d29b7d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/2485e3c30ba047b2b60cacacc9d29b7d 2023-04-08 02:56:43,259 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/6cfa536c55594bdabb40010912698049 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/6cfa536c55594bdabb40010912698049 2023-04-08 02:56:43,259 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/a54a66c1701f4d94b46554322126fa6b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/a54a66c1701f4d94b46554322126fa6b 2023-04-08 02:56:43,260 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/dd2591ebd6c44801a5566da8d1048c3b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/dd2591ebd6c44801a5566da8d1048c3b 2023-04-08 02:56:43,261 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/c4a9e0f392ba47b9b7bc3582907915c6 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/c4a9e0f392ba47b9b7bc3582907915c6 2023-04-08 02:56:43,262 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/a146727b257547aead830f6b888f312b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/a146727b257547aead830f6b888f312b 2023-04-08 02:56:43,263 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/2bbb486a8a724dfb83a301a9a0790c56 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/2bbb486a8a724dfb83a301a9a0790c56 2023-04-08 02:56:43,263 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/544beee08dfc4e37ad66220eafdccce8 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/544beee08dfc4e37ad66220eafdccce8 2023-04-08 02:56:43,264 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/6e04514b4b1345358b906cd8f377e834 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/6e04514b4b1345358b906cd8f377e834 2023-04-08 02:56:43,265 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/18e7df7da9494063a0d30dee4f9e185f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/18e7df7da9494063a0d30dee4f9e185f 2023-04-08 02:56:43,266 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/42842c03d3bd4d928e48b5ff4e165ec2 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/42842c03d3bd4d928e48b5ff4e165ec2 2023-04-08 02:56:43,267 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/bfc7b7bfccc244708ecccbe839169248 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/bfc7b7bfccc244708ecccbe839169248 2023-04-08 02:56:43,267 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/0cd178279194400f98433c18bb1420e8 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/0cd178279194400f98433c18bb1420e8 2023-04-08 02:56:43,268 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/d516fb12371945ab9580fb3bc866127b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/d516fb12371945ab9580fb3bc866127b 2023-04-08 02:56:43,269 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/f10b0966e3d146eaa746e6de3543500c to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/f10b0966e3d146eaa746e6de3543500c 2023-04-08 02:56:43,272 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/d62c465e7adf46a28719aed70915c5a1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/448b524cc31445aeab7d537e7afbb156, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/f58e638146f94d9daab7142c722f93b2, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/53fb381951a0431baf4c122852dc4c6d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/0fe03742967447ff918a37718e65e1de, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/31753b646c1c4d77ac26de8ffe8d614b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/c537fa9f84c94493bcf8ffc2d0c6ff61, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/e66558ba4c1b4bb2b4bfbd92fced47de, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/919d2585676d45e9b911d664023d5fdd, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/2287672a7c114fa595e115aa7a4b35ce, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/8114c671901b47c5a55bb29066daf063, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/dbb6ea05f4f34015aa28c353d6e34347, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/b907ccacf17547bdae4390659a7cf622, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/16b5beffcd694d7ea1679b6d773f305d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/688d702440754c58a86acd68cec74032, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/bade4b55b34a4875a26af36be0514c73, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/a9f10714b9b1436cbb71a132c56ed159, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/dd6f598366374a00a2c4f51fae2d4c9c, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/9e93b512b141450bae610114097f33c1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/64ee3ac00f054e36b15b4138781b88ca, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/eb3c96c14b8c466094def0a464eef38e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/7284cc7dbc06481f90bc175ee8a0d91e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/c35cb67497d24afba40d9e8da20b6d60, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/65a6b8c0cbaa4f748d82ac909abfe8be, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/eecaed1b39d44b8da8bb2601cac402dc, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/7fed4017f811488896b29200b5150fe6, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/58377c872ad44a15960cf8ee36c94085, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/47c44a7c021e40c0a99c830f82143464, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/c7f8ce256513403a9958bf3fae4c413e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/447aee9245c84aa1bd79851be90bec5a, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/230dd3a209c14946874c18137ad17815, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/75c113a2e6e14be69a12eb8ade7dc675, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/1758a206e2ae45bba8b21c06921a12e6, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/454c6df2ab7243f68776a1facda42272, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/3f345fd87d5b49fc9e84d99853bbd643, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/50aded6ea07042e79c3827d27790607e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/ecdaba59d8474dd79738fd5c0e0b3e23, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/bb3035d73e1949098981ccdbfc532882] to archive 2023-04-08 02:56:43,272 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 02:56:43,273 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/d62c465e7adf46a28719aed70915c5a1 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/d62c465e7adf46a28719aed70915c5a1 2023-04-08 02:56:43,274 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/448b524cc31445aeab7d537e7afbb156 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/448b524cc31445aeab7d537e7afbb156 2023-04-08 02:56:43,275 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/f58e638146f94d9daab7142c722f93b2 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/f58e638146f94d9daab7142c722f93b2 2023-04-08 02:56:43,276 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/53fb381951a0431baf4c122852dc4c6d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/53fb381951a0431baf4c122852dc4c6d 2023-04-08 02:56:43,276 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/0fe03742967447ff918a37718e65e1de to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/0fe03742967447ff918a37718e65e1de 2023-04-08 02:56:43,277 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/31753b646c1c4d77ac26de8ffe8d614b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/31753b646c1c4d77ac26de8ffe8d614b 2023-04-08 02:56:43,278 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/c537fa9f84c94493bcf8ffc2d0c6ff61 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/c537fa9f84c94493bcf8ffc2d0c6ff61 2023-04-08 02:56:43,279 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/e66558ba4c1b4bb2b4bfbd92fced47de to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/e66558ba4c1b4bb2b4bfbd92fced47de 2023-04-08 02:56:43,279 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/919d2585676d45e9b911d664023d5fdd to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/919d2585676d45e9b911d664023d5fdd 2023-04-08 02:56:43,280 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/2287672a7c114fa595e115aa7a4b35ce to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/2287672a7c114fa595e115aa7a4b35ce 2023-04-08 02:56:43,281 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/8114c671901b47c5a55bb29066daf063 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/8114c671901b47c5a55bb29066daf063 2023-04-08 02:56:43,282 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/dbb6ea05f4f34015aa28c353d6e34347 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/dbb6ea05f4f34015aa28c353d6e34347 2023-04-08 02:56:43,283 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/b907ccacf17547bdae4390659a7cf622 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/b907ccacf17547bdae4390659a7cf622 2023-04-08 02:56:43,283 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/16b5beffcd694d7ea1679b6d773f305d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/16b5beffcd694d7ea1679b6d773f305d 2023-04-08 02:56:43,284 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/688d702440754c58a86acd68cec74032 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/688d702440754c58a86acd68cec74032 2023-04-08 02:56:43,285 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/bade4b55b34a4875a26af36be0514c73 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/bade4b55b34a4875a26af36be0514c73 2023-04-08 02:56:43,286 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/a9f10714b9b1436cbb71a132c56ed159 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/a9f10714b9b1436cbb71a132c56ed159 2023-04-08 02:56:43,287 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/dd6f598366374a00a2c4f51fae2d4c9c to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/dd6f598366374a00a2c4f51fae2d4c9c 2023-04-08 02:56:43,287 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/9e93b512b141450bae610114097f33c1 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/9e93b512b141450bae610114097f33c1 2023-04-08 02:56:43,288 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/64ee3ac00f054e36b15b4138781b88ca to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/64ee3ac00f054e36b15b4138781b88ca 2023-04-08 02:56:43,289 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/eb3c96c14b8c466094def0a464eef38e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/eb3c96c14b8c466094def0a464eef38e 2023-04-08 02:56:43,290 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/7284cc7dbc06481f90bc175ee8a0d91e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/7284cc7dbc06481f90bc175ee8a0d91e 2023-04-08 02:56:43,291 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/c35cb67497d24afba40d9e8da20b6d60 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/c35cb67497d24afba40d9e8da20b6d60 2023-04-08 02:56:43,292 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/65a6b8c0cbaa4f748d82ac909abfe8be to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/65a6b8c0cbaa4f748d82ac909abfe8be 2023-04-08 02:56:43,293 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/eecaed1b39d44b8da8bb2601cac402dc to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/eecaed1b39d44b8da8bb2601cac402dc 2023-04-08 02:56:43,294 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/7fed4017f811488896b29200b5150fe6 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/7fed4017f811488896b29200b5150fe6 2023-04-08 02:56:43,294 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/58377c872ad44a15960cf8ee36c94085 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/58377c872ad44a15960cf8ee36c94085 2023-04-08 02:56:43,295 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/47c44a7c021e40c0a99c830f82143464 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/47c44a7c021e40c0a99c830f82143464 2023-04-08 02:56:43,296 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/c7f8ce256513403a9958bf3fae4c413e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/c7f8ce256513403a9958bf3fae4c413e 2023-04-08 02:56:43,297 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/447aee9245c84aa1bd79851be90bec5a to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/447aee9245c84aa1bd79851be90bec5a 2023-04-08 02:56:43,298 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/230dd3a209c14946874c18137ad17815 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/230dd3a209c14946874c18137ad17815 2023-04-08 02:56:43,299 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/75c113a2e6e14be69a12eb8ade7dc675 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/75c113a2e6e14be69a12eb8ade7dc675 2023-04-08 02:56:43,300 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/1758a206e2ae45bba8b21c06921a12e6 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/1758a206e2ae45bba8b21c06921a12e6 2023-04-08 02:56:43,301 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/454c6df2ab7243f68776a1facda42272 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/454c6df2ab7243f68776a1facda42272 2023-04-08 02:56:43,301 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/3f345fd87d5b49fc9e84d99853bbd643 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/3f345fd87d5b49fc9e84d99853bbd643 2023-04-08 02:56:43,302 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/50aded6ea07042e79c3827d27790607e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/50aded6ea07042e79c3827d27790607e 2023-04-08 02:56:43,303 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/ecdaba59d8474dd79738fd5c0e0b3e23 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/ecdaba59d8474dd79738fd5c0e0b3e23 2023-04-08 02:56:43,304 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/bb3035d73e1949098981ccdbfc532882 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/bb3035d73e1949098981ccdbfc532882 2023-04-08 02:56:43,306 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] regionserver.HStore(2712): Moving the files [hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/45b995b0b41a422fa66bab9f110fc233, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/8fabc2dfdfc24b7a838160f26036567e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/112b5fc443b043db8b34b22313a42ff7, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/7a58b6eadcd94426926df15b6ea55587, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/6ef17dd2bd4c44d089c44fcc2492035f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/4c3952ef594543229ece4804f89cce9e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/7c558cbd014543658c38040d133718d6, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/bb267508ba034d348146a4bc32cb6091, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/a5e8a0d098f944d8815ddeaff0f6973e, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/25ec35eb369c47e0a2eaddef8f5a346d, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/558b7d7471224163a60dd4e6e30a34eb, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/3b0affee2c7b4ff2808d951de9fef160, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/14cb6dab080b41cba5f417d7faf51f00, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/51524096644e44bbb5d3095f857ed383, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/350c2a97fe214687ba7668954a74e352, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/0720a564ca8e48eebadb9e059e331bd9, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/42a545fd2fc94998a0da12cea2797844, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/05c6a06e18ac4a5fa6c074f361737086, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/2e2e344230f6425e9e87cb61a4fd2f81, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/0c711a7cd18e4c99a3fa13aca596d652, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/dcf1503d625540dca4c4eb9f70d53178, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/cd13504f803c473484f1bbb32605048f, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/bbf62a9b766a4b05bd9507489a83f635, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/c5c5655b56914dd68f201e66a098118a, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/18d1b2d2eaa54600bcd657749a006d4b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/9044562768224551825fd54fa6d77df1, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/c0319156bd0542f08054e1121fa3882b, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/81743c2d325a418abfb9306be9485e61, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/327448e0c7fc43528e927a2b21699d0a, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/1dd126b1990a435187d49bf5d0f953e4, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/a7731c79a7334ee49a59f82614d35486, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/42172418da164ca3a1bad08b192f4e45, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/51b9d3d645e348e7857332d51e5522bf, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/265a46bb9fca46a8af3c931db3922279, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/6e5c88c093ca4f73a36d80b613aa3116, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/25cb943541014d6d9b52e9419a73c178, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/9ae64b1bf8544423bad76cecf3bab60c] to archive 2023-04-08 02:56:43,307 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(360): Archiving compacted files. 2023-04-08 02:56:43,308 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/45b995b0b41a422fa66bab9f110fc233 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/45b995b0b41a422fa66bab9f110fc233 2023-04-08 02:56:43,309 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/8fabc2dfdfc24b7a838160f26036567e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/8fabc2dfdfc24b7a838160f26036567e 2023-04-08 02:56:43,310 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/112b5fc443b043db8b34b22313a42ff7 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/112b5fc443b043db8b34b22313a42ff7 2023-04-08 02:56:43,310 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/7a58b6eadcd94426926df15b6ea55587 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/7a58b6eadcd94426926df15b6ea55587 2023-04-08 02:56:43,311 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/6ef17dd2bd4c44d089c44fcc2492035f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/6ef17dd2bd4c44d089c44fcc2492035f 2023-04-08 02:56:43,312 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/4c3952ef594543229ece4804f89cce9e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/4c3952ef594543229ece4804f89cce9e 2023-04-08 02:56:43,313 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/7c558cbd014543658c38040d133718d6 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/7c558cbd014543658c38040d133718d6 2023-04-08 02:56:43,313 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/bb267508ba034d348146a4bc32cb6091 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/bb267508ba034d348146a4bc32cb6091 2023-04-08 02:56:43,314 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/a5e8a0d098f944d8815ddeaff0f6973e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/a5e8a0d098f944d8815ddeaff0f6973e 2023-04-08 02:56:43,315 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/25ec35eb369c47e0a2eaddef8f5a346d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/25ec35eb369c47e0a2eaddef8f5a346d 2023-04-08 02:56:43,315 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/558b7d7471224163a60dd4e6e30a34eb to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/558b7d7471224163a60dd4e6e30a34eb 2023-04-08 02:56:43,316 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/3b0affee2c7b4ff2808d951de9fef160 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/3b0affee2c7b4ff2808d951de9fef160 2023-04-08 02:56:43,317 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/14cb6dab080b41cba5f417d7faf51f00 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/14cb6dab080b41cba5f417d7faf51f00 2023-04-08 02:56:43,317 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/51524096644e44bbb5d3095f857ed383 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/51524096644e44bbb5d3095f857ed383 2023-04-08 02:56:43,318 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/350c2a97fe214687ba7668954a74e352 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/350c2a97fe214687ba7668954a74e352 2023-04-08 02:56:43,319 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/0720a564ca8e48eebadb9e059e331bd9 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/0720a564ca8e48eebadb9e059e331bd9 2023-04-08 02:56:43,320 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/42a545fd2fc94998a0da12cea2797844 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/42a545fd2fc94998a0da12cea2797844 2023-04-08 02:56:43,320 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/05c6a06e18ac4a5fa6c074f361737086 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/05c6a06e18ac4a5fa6c074f361737086 2023-04-08 02:56:43,321 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/2e2e344230f6425e9e87cb61a4fd2f81 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/2e2e344230f6425e9e87cb61a4fd2f81 2023-04-08 02:56:43,322 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/0c711a7cd18e4c99a3fa13aca596d652 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/0c711a7cd18e4c99a3fa13aca596d652 2023-04-08 02:56:43,323 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/dcf1503d625540dca4c4eb9f70d53178 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/dcf1503d625540dca4c4eb9f70d53178 2023-04-08 02:56:43,324 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/cd13504f803c473484f1bbb32605048f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/cd13504f803c473484f1bbb32605048f 2023-04-08 02:56:43,325 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/bbf62a9b766a4b05bd9507489a83f635 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/bbf62a9b766a4b05bd9507489a83f635 2023-04-08 02:56:43,325 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/c5c5655b56914dd68f201e66a098118a to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/c5c5655b56914dd68f201e66a098118a 2023-04-08 02:56:43,326 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/18d1b2d2eaa54600bcd657749a006d4b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/18d1b2d2eaa54600bcd657749a006d4b 2023-04-08 02:56:43,327 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/9044562768224551825fd54fa6d77df1 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/9044562768224551825fd54fa6d77df1 2023-04-08 02:56:43,328 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/c0319156bd0542f08054e1121fa3882b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/c0319156bd0542f08054e1121fa3882b 2023-04-08 02:56:43,329 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/81743c2d325a418abfb9306be9485e61 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/81743c2d325a418abfb9306be9485e61 2023-04-08 02:56:43,330 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/327448e0c7fc43528e927a2b21699d0a to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/327448e0c7fc43528e927a2b21699d0a 2023-04-08 02:56:43,330 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/1dd126b1990a435187d49bf5d0f953e4 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/1dd126b1990a435187d49bf5d0f953e4 2023-04-08 02:56:43,331 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/a7731c79a7334ee49a59f82614d35486 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/a7731c79a7334ee49a59f82614d35486 2023-04-08 02:56:43,332 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/42172418da164ca3a1bad08b192f4e45 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/42172418da164ca3a1bad08b192f4e45 2023-04-08 02:56:43,333 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/51b9d3d645e348e7857332d51e5522bf to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/51b9d3d645e348e7857332d51e5522bf 2023-04-08 02:56:43,334 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/265a46bb9fca46a8af3c931db3922279 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/265a46bb9fca46a8af3c931db3922279 2023-04-08 02:56:43,335 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/6e5c88c093ca4f73a36d80b613aa3116 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/6e5c88c093ca4f73a36d80b613aa3116 2023-04-08 02:56:43,336 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/25cb943541014d6d9b52e9419a73c178 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/25cb943541014d6d9b52e9419a73c178 2023-04-08 02:56:43,337 DEBUG [StoreCloser-TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.-1] backup.HFileArchiver(582): Archived from FileableStoreFile, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/9ae64b1bf8544423bad76cecf3bab60c to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/9ae64b1bf8544423bad76cecf3bab60c 2023-04-08 02:56:43,340 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/recovered.edits/1022.seqid, newMaxSeqId=1022, maxSeqId=4 2023-04-08 02:56:43,341 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b. 2023-04-08 02:56:43,341 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 337c881227bdfa873bb568857a6d7e4b: 2023-04-08 02:56:43,343 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.UnassignRegionHandler(147): Closed 337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:43,343 INFO [PEWorker-2] assignment.RegionStateStore(219): pid=63 updating hbase:meta row=337c881227bdfa873bb568857a6d7e4b, regionState=CLOSED 2023-04-08 02:56:43,343 DEBUG [PEWorker-2] assignment.RegionStateStore(405): Put {"totalColumns":2,"row":"TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.","families":{"info":[{"qualifier":"regioninfo","vlen":52,"tag":[],"timestamp":"1680922603343"},{"qualifier":"state","vlen":6,"tag":[],"timestamp":"1680922603343"}]},"ts":"1680922603343"} 2023-04-08 02:56:43,345 INFO [PEWorker-2] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=64, resume processing ppid=63 2023-04-08 02:56:43,345 INFO [PEWorker-2] procedure2.ProcedureExecutor(1410): Finished pid=64, ppid=63, state=SUCCESS; CloseRegionProcedure 337c881227bdfa873bb568857a6d7e4b, server=jenkins-hbase4.apache.org,35089,1680922417983 in 1.9270 sec 2023-04-08 02:56:43,346 INFO [PEWorker-3] procedure2.ProcedureExecutor(1824): Finished subprocedure pid=63, resume processing ppid=62 2023-04-08 02:56:43,346 INFO [PEWorker-3] procedure2.ProcedureExecutor(1410): Finished pid=63, ppid=62, state=SUCCESS; TransitRegionStateProcedure table=TestAcidGuarantees, region=337c881227bdfa873bb568857a6d7e4b, UNASSIGN in 1.9300 sec 2023-04-08 02:56:43,346 DEBUG [PEWorker-5] hbase.MetaTableAccessor(2093): Put {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":2,"tag":[],"timestamp":"1680922603346"}]},"ts":"1680922603346"} 2023-04-08 02:56:43,347 INFO [PEWorker-5] hbase.MetaTableAccessor(1635): Updated tableName=TestAcidGuarantees, state=DISABLED in hbase:meta 2023-04-08 02:56:43,348 INFO [PEWorker-5] procedure.DisableTableProcedure(305): Set TestAcidGuarantees to state=DISABLED 2023-04-08 02:56:43,349 INFO [PEWorker-5] procedure2.ProcedureExecutor(1410): Finished pid=62, state=SUCCESS; DisableTableProcedure table=TestAcidGuarantees in 1.9400 sec 2023-04-08 02:56:43,516 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=62 2023-04-08 02:56:43,516 INFO [Listener at localhost/39741] client.HBaseAdmin$TableFuture(3541): Operation: DISABLE, Table Name: default:TestAcidGuarantees, procId: 62 completed 2023-04-08 02:56:43,517 INFO [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.HMaster$5(2228): Client=jenkins//172.31.14.131 delete TestAcidGuarantees 2023-04-08 02:56:43,517 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] procedure2.ProcedureExecutor(1029): Stored pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:56:43,519 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(101): Waiting for RIT for pid=65, state=RUNNABLE:DELETE_TABLE_PRE_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:56:43,519 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-08 02:56:43,519 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(113): Deleting regions from filesystem for pid=65, state=RUNNABLE:DELETE_TABLE_CLEAR_FS_LAYOUT, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:56:43,522 DEBUG [HFileArchiver-13] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:43,524 DEBUG [HFileArchiver-13] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A, FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B, FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C, FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/recovered.edits] 2023-04-08 02:56:43,526 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/52ac88894a7743119903a24ecab900cc to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/52ac88894a7743119903a24ecab900cc 2023-04-08 02:56:43,527 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/78eeab7a694840d2988fc6d886ade9e9 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/78eeab7a694840d2988fc6d886ade9e9 2023-04-08 02:56:43,528 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/986c155ef0d14dd6bfa957d02a4f9294 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/A/986c155ef0d14dd6bfa957d02a4f9294 2023-04-08 02:56:43,530 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/22b139cb3a3b4bc69d051bd20fa86a30 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/22b139cb3a3b4bc69d051bd20fa86a30 2023-04-08 02:56:43,531 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/e74e69ecc7114b0ba721c921ea86505c to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/e74e69ecc7114b0ba721c921ea86505c 2023-04-08 02:56:43,532 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/f473a2d6f0ed429e95fa9425f292b961 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/B/f473a2d6f0ed429e95fa9425f292b961 2023-04-08 02:56:43,533 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/02e7dade70de45508852d7fb8e5574dc to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/02e7dade70de45508852d7fb8e5574dc 2023-04-08 02:56:43,534 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/7bfdffabe2c24952aa34b60bb10225fd to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/7bfdffabe2c24952aa34b60bb10225fd 2023-04-08 02:56:43,534 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/9d0ab4a6fcdc40168cdf09021f6ff464 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/C/9d0ab4a6fcdc40168cdf09021f6ff464 2023-04-08 02:56:43,536 DEBUG [HFileArchiver-13] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/recovered.edits/1022.seqid to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b/recovered.edits/1022.seqid 2023-04-08 02:56:43,536 DEBUG [HFileArchiver-13] backup.HFileArchiver(596): Deleted hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/.tmp/data/default/TestAcidGuarantees/337c881227bdfa873bb568857a6d7e4b 2023-04-08 02:56:43,536 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(328): Archived TestAcidGuarantees regions 2023-04-08 02:56:43,537 DEBUG [PEWorker-1] backup.HFileArchiver(131): ARCHIVING hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-08 02:56:43,537 DEBUG [PEWorker-1] backup.HFileArchiver(159): Archiving [FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A] 2023-04-08 02:56:43,540 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408058a16854272472fa6ba21fd0a44a25d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408058a16854272472fa6ba21fd0a44a25d 2023-04-08 02:56:43,541 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304080acec8c50cba461fa1dc12b0841600d8 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304080acec8c50cba461fa1dc12b0841600d8 2023-04-08 02:56:43,541 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304081e4d045903c9465189e15c0fbc77e10d to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304081e4d045903c9465189e15c0fbc77e10d 2023-04-08 02:56:43,542 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408204724480c914ff98baac8bcbfb691d3 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408204724480c914ff98baac8bcbfb691d3 2023-04-08 02:56:43,543 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408221d61dbdfca45518613025137a4a541 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408221d61dbdfca45518613025137a4a541 2023-04-08 02:56:43,543 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304082ceac21d52e643159fcda723f3201cfb to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304082ceac21d52e643159fcda723f3201cfb 2023-04-08 02:56:43,544 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408315b92ec38b3428e92b2b44bbf87c752 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408315b92ec38b3428e92b2b44bbf87c752 2023-04-08 02:56:43,545 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304084001c7fa447b4cd6a03d9f77029f02a8 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304084001c7fa447b4cd6a03d9f77029f02a8 2023-04-08 02:56:43,546 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408430dce0c677b4c919714656a4ef1b59c to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408430dce0c677b4c919714656a4ef1b59c 2023-04-08 02:56:43,547 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040854a91fb8b8fd4a1c9038296b4dff0239 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040854a91fb8b8fd4a1c9038296b4dff0239 2023-04-08 02:56:43,547 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304085682a570759748a2a20a17da58749b76 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304085682a570759748a2a20a17da58749b76 2023-04-08 02:56:43,548 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408683c3fe99bf74cdcb163a7609264053c to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408683c3fe99bf74cdcb163a7609264053c 2023-04-08 02:56:43,549 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040878f1b83d618c4f12ae2afed0f086a860 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040878f1b83d618c4f12ae2afed0f086a860 2023-04-08 02:56:43,550 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304087e723c6d792742dfa6e5b7e9b18ababd to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e202304087e723c6d792742dfa6e5b7e9b18ababd 2023-04-08 02:56:43,550 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040886fd1993efc54a0a94b06c5fac71018b to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040886fd1993efc54a0a94b06c5fac71018b 2023-04-08 02:56:43,551 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040890e9cd149fcd4e7ca6274a55f3c211b8 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e2023040890e9cd149fcd4e7ca6274a55f3c211b8 2023-04-08 02:56:43,552 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408abb64fb8ceb04cb891c9b41cf2bf4b63 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408abb64fb8ceb04cb891c9b41cf2bf4b63 2023-04-08 02:56:43,553 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408aed15bb58d984994b642562305aef6af to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408aed15bb58d984994b642562305aef6af 2023-04-08 02:56:43,554 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408b27c1222a0384ad1a32cf58162e84a2f to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408b27c1222a0384ad1a32cf58162e84a2f 2023-04-08 02:56:43,555 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408bd060db7b4384a2eb84c5b42abcfe309 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408bd060db7b4384a2eb84c5b42abcfe309 2023-04-08 02:56:43,556 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408beed71896a1c49ceb04253821875e855 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408beed71896a1c49ceb04253821875e855 2023-04-08 02:56:43,557 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408c3a618c6a790409a9cb20ecbe2bd9fb6 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408c3a618c6a790409a9cb20ecbe2bd9fb6 2023-04-08 02:56:43,558 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408c870721823ab453c97700805ab10fcaa to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408c870721823ab453c97700805ab10fcaa 2023-04-08 02:56:43,559 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408c910b35944d7473082d05bb188ea87a2 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408c910b35944d7473082d05bb188ea87a2 2023-04-08 02:56:43,560 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408cd8c78523b384bd4a14e10b54fe23d78 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408cd8c78523b384bd4a14e10b54fe23d78 2023-04-08 02:56:43,560 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408cdf63845527c454d9cc6a97d6d3acf37 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408cdf63845527c454d9cc6a97d6d3acf37 2023-04-08 02:56:43,561 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408cf996f4ab5dd4ca3bf398cb0b29a3a83 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408cf996f4ab5dd4ca3bf398cb0b29a3a83 2023-04-08 02:56:43,562 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408d7c200d87f3e46f7ad57cb0abf81aab7 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408d7c200d87f3e46f7ad57cb0abf81aab7 2023-04-08 02:56:43,563 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408e04b5d52fa57477080fcaa461a69642e to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408e04b5d52fa57477080fcaa461a69642e 2023-04-08 02:56:43,564 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408f6c00a4d35514f7f9cc6b7e8fdc0c472 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408f6c00a4d35514f7f9cc6b7e8fdc0c472 2023-04-08 02:56:43,564 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408fbf3b3224ab54f9883f2440764ead719 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408fbf3b3224ab54f9883f2440764ead719 2023-04-08 02:56:43,565 DEBUG [PEWorker-1] backup.HFileArchiver(582): Archived from FileablePath, hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408fea0f27d30ea4ecab7879d4aa78f91e0 to hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/archive/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3/A/d41d8cd98f00b204e9800998ecf8427e20230408fea0f27d30ea4ecab7879d4aa78f91e0 2023-04-08 02:56:43,565 DEBUG [PEWorker-1] backup.HFileArchiver(596): Deleted hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/mobdir/data/default/TestAcidGuarantees/6995762b1b921fda31b7b26a0c5785c3 2023-04-08 02:56:43,567 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(118): Deleting regions from META for pid=65, state=RUNNABLE:DELETE_TABLE_REMOVE_FROM_META, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:56:43,568 WARN [PEWorker-1] procedure.DeleteTableProcedure(384): Deleting some vestigial 1 rows of TestAcidGuarantees from hbase:meta 2023-04-08 02:56:43,569 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(421): Removing 'TestAcidGuarantees' descriptor. 2023-04-08 02:56:43,570 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(124): Deleting assignment state for pid=65, state=RUNNABLE:DELETE_TABLE_UNASSIGN_REGIONS, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:56:43,570 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(411): Removing 'TestAcidGuarantees' from region states. 2023-04-08 02:56:43,570 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.","families":{"info":[{"qualifier":"","vlen":0,"tag":[],"timestamp":"1680922603570"}]},"ts":"9223372036854775807"} 2023-04-08 02:56:43,571 INFO [PEWorker-1] hbase.MetaTableAccessor(1788): Deleted 1 regions from META 2023-04-08 02:56:43,571 DEBUG [PEWorker-1] hbase.MetaTableAccessor(1789): Deleted regions: [{ENCODED => 337c881227bdfa873bb568857a6d7e4b, NAME => 'TestAcidGuarantees,,1680922574360.337c881227bdfa873bb568857a6d7e4b.', STARTKEY => '', ENDKEY => ''}] 2023-04-08 02:56:43,572 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(415): Marking 'TestAcidGuarantees' as deleted. 2023-04-08 02:56:43,572 DEBUG [PEWorker-1] hbase.MetaTableAccessor(2093): Delete {"totalColumns":1,"row":"TestAcidGuarantees","families":{"table":[{"qualifier":"state","vlen":0,"tag":[],"timestamp":"1680922603572"}]},"ts":"9223372036854775807"} 2023-04-08 02:56:43,573 INFO [PEWorker-1] hbase.MetaTableAccessor(1658): Deleted table TestAcidGuarantees state from META 2023-04-08 02:56:43,575 DEBUG [PEWorker-1] procedure.DeleteTableProcedure(130): Finished pid=65, state=RUNNABLE:DELETE_TABLE_POST_OPERATION, locked=true; DeleteTableProcedure table=TestAcidGuarantees 2023-04-08 02:56:43,575 INFO [PEWorker-1] procedure2.ProcedureExecutor(1410): Finished pid=65, state=SUCCESS; DeleteTableProcedure table=TestAcidGuarantees in 58 msec 2023-04-08 02:56:43,620 DEBUG [RpcServer.default.FPBQ.Fifo.handler=2,queue=0,port=46135] master.MasterRpcServices(1227): Checking to see if procedure is done pid=65 2023-04-08 02:56:43,620 INFO [Listener at localhost/39741] client.HBaseAdmin$TableFuture(3541): Operation: DELETE, Table Name: default:TestAcidGuarantees, procId: 65 completed 2023-04-08 02:56:43,632 INFO [Listener at localhost/39741] hbase.ResourceChecker(175): after: TestAcidGuaranteesWithEagerPolicy#testMobGetAtomicity Thread=228 (was 232), OpenFileDescriptor=575 (was 577), MaxFileDescriptor=60000 (was 60000), SystemLoadAverage=370 (was 424), ProcessCount=172 (was 172), AvailableMemoryMB=1725 (was 1821) 2023-04-08 02:56:43,632 INFO [Listener at localhost/39741] hbase.HBaseTestingUtility(1286): Shutting down minicluster 2023-04-08 02:56:43,632 INFO [Listener at localhost/39741] client.ConnectionImplementation(1974): Closing master protocol: MasterService 2023-04-08 02:56:43,633 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ed65498 to 127.0.0.1:64826 2023-04-08 02:56:43,633 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:56:43,633 DEBUG [Listener at localhost/39741] util.JVMClusterUtil(237): Shutting down HBase Cluster 2023-04-08 02:56:43,633 DEBUG [Listener at localhost/39741] util.JVMClusterUtil(257): Found active master hash=1335016385, stopped=false 2023-04-08 02:56:43,633 INFO [Listener at localhost/39741] master.ServerManager(901): Cluster shutdown requested of master=jenkins-hbase4.apache.org,46135,1680922416713 2023-04-08 02:56:43,634 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-08 02:56:43,634 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/running 2023-04-08 02:56:43,634 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-08 02:56:43,634 INFO [Listener at localhost/39741] procedure2.ProcedureExecutor(629): Stopping 2023-04-08 02:56:43,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-08 02:56:43,635 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/running 2023-04-08 02:56:43,635 DEBUG [Listener at localhost/39741] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x6ebbc202 to 127.0.0.1:64826 2023-04-08 02:56:43,635 DEBUG [Listener at localhost/39741] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:56:43,635 INFO [Listener at localhost/39741] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,35089,1680922417983' ***** 2023-04-08 02:56:43,635 INFO [Listener at localhost/39741] regionserver.HRegionServer(2310): STOPPED: Shutdown requested 2023-04-08 02:56:43,636 INFO [RS:0;jenkins-hbase4:35089] regionserver.HeapMemoryManager(220): Stopping 2023-04-08 02:56:43,636 INFO [RS:0;jenkins-hbase4:35089] flush.RegionServerFlushTableProcedureManager(117): Stopping region server flush procedure manager gracefully. 2023-04-08 02:56:43,636 INFO [MemStoreFlusher.0] regionserver.MemStoreFlusher$FlushHandler(361): MemStoreFlusher.0 exiting 2023-04-08 02:56:43,636 INFO [RS:0;jenkins-hbase4:35089] snapshot.RegionServerSnapshotManager(137): Stopping RegionServerSnapshotManager gracefully. 2023-04-08 02:56:43,636 INFO [RS:0;jenkins-hbase4:35089] regionserver.HRegionServer(3304): Received CLOSE for 087fd945a79ca5b739610704cbfa7019 2023-04-08 02:56:43,636 INFO [RS:0;jenkins-hbase4:35089] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:43,636 DEBUG [RS:0;jenkins-hbase4:35089] zookeeper.ReadOnlyZKClient(361): Close zookeeper connection 0x7e8c7836 to 127.0.0.1:64826 2023-04-08 02:56:43,637 DEBUG [RS:0;jenkins-hbase4:35089] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:56:43,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 087fd945a79ca5b739610704cbfa7019, disabling compactions & flushes 2023-04-08 02:56:43,637 INFO [RS:0;jenkins-hbase4:35089] regionserver.CompactSplit(434): Waiting for Split Thread to finish... 2023-04-08 02:56:43,637 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:namespace,,1680922421094.087fd945a79ca5b739610704cbfa7019. 2023-04-08 02:56:43,637 INFO [RS:0;jenkins-hbase4:35089] regionserver.CompactSplit(434): Waiting for Large Compaction Thread to finish... 2023-04-08 02:56:43,637 INFO [RS:0;jenkins-hbase4:35089] regionserver.CompactSplit(434): Waiting for Small Compaction Thread to finish... 2023-04-08 02:56:43,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:namespace,,1680922421094.087fd945a79ca5b739610704cbfa7019. 2023-04-08 02:56:43,637 INFO [RS:0;jenkins-hbase4:35089] regionserver.HRegionServer(3304): Received CLOSE for 1588230740 2023-04-08 02:56:43,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:namespace,,1680922421094.087fd945a79ca5b739610704cbfa7019. after waiting 0 ms 2023-04-08 02:56:43,637 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:namespace,,1680922421094.087fd945a79ca5b739610704cbfa7019. 2023-04-08 02:56:43,637 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 087fd945a79ca5b739610704cbfa7019 1/1 column families, dataSize=78 B heapSize=488 B 2023-04-08 02:56:43,637 INFO [RS:0;jenkins-hbase4:35089] regionserver.HRegionServer(1475): Waiting on 2 regions to close 2023-04-08 02:56:43,638 DEBUG [RS:0;jenkins-hbase4:35089] regionserver.HRegionServer(1479): Online Regions={1588230740=hbase:meta,,1.1588230740, 087fd945a79ca5b739610704cbfa7019=hbase:namespace,,1680922421094.087fd945a79ca5b739610704cbfa7019.} 2023-04-08 02:56:43,638 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1604): Closing 1588230740, disabling compactions & flushes 2023-04-08 02:56:43,638 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1626): Closing region hbase:meta,,1.1588230740 2023-04-08 02:56:43,638 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1647): Waiting without time limit for close lock on hbase:meta,,1.1588230740 2023-04-08 02:56:43,638 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1714): Acquired close lock on hbase:meta,,1.1588230740 after waiting 0 ms 2023-04-08 02:56:43,638 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1724): Updates disabled for region hbase:meta,,1.1588230740 2023-04-08 02:56:43,638 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2745): Flushing 1588230740 3/3 column families, dataSize=20.92 KB heapSize=36.24 KB 2023-04-08 02:56:43,640 DEBUG [RS:0;jenkins-hbase4:35089] regionserver.HRegionServer(1505): Waiting on 087fd945a79ca5b739610704cbfa7019, 1588230740 2023-04-08 02:56:43,659 INFO [regionserver/jenkins-hbase4:0.leaseChecker] regionserver.LeaseManager(133): Closed leases 2023-04-08 02:56:43,752 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: MemstoreFlusherChore was stopped 2023-04-08 02:56:43,752 INFO [regionserver/jenkins-hbase4:0.Chore.1] hbase.ScheduledChore(146): Chore: CompactionChecker was stopped 2023-04-08 02:56:43,840 DEBUG [RS:0;jenkins-hbase4:35089] regionserver.HRegionServer(1505): Waiting on 087fd945a79ca5b739610704cbfa7019, 1588230740 2023-04-08 02:56:44,040 DEBUG [RS:0;jenkins-hbase4:35089] regionserver.HRegionServer(1505): Waiting on 087fd945a79ca5b739610704cbfa7019, 1588230740 2023-04-08 02:56:44,046 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=18.26 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/1588230740/.tmp/info/90b32090e52d4fd29067ee8baa366149 2023-04-08 02:56:44,050 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 90b32090e52d4fd29067ee8baa366149 2023-04-08 02:56:44,052 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=78 B at sequenceid=6 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/namespace/087fd945a79ca5b739610704cbfa7019/.tmp/info/2425bf81fe5b463a9671412d5d825044 2023-04-08 02:56:44,055 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=588 B at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/1588230740/.tmp/rep_barrier/78157665902b43cbb0f9f5a1708572ea 2023-04-08 02:56:44,056 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/namespace/087fd945a79ca5b739610704cbfa7019/.tmp/info/2425bf81fe5b463a9671412d5d825044 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/namespace/087fd945a79ca5b739610704cbfa7019/info/2425bf81fe5b463a9671412d5d825044 2023-04-08 02:56:44,057 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 78157665902b43cbb0f9f5a1708572ea 2023-04-08 02:56:44,059 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/namespace/087fd945a79ca5b739610704cbfa7019/info/2425bf81fe5b463a9671412d5d825044, entries=2, sequenceid=6, filesize=4.8 K 2023-04-08 02:56:44,060 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~78 B/78, heapSize ~472 B/472, currentSize=0 B/0 for 087fd945a79ca5b739610704cbfa7019 in 423ms, sequenceid=6, compaction requested=false 2023-04-08 02:56:44,066 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/namespace/087fd945a79ca5b739610704cbfa7019/recovered.edits/9.seqid, newMaxSeqId=9, maxSeqId=1 2023-04-08 02:56:44,067 INFO [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:namespace,,1680922421094.087fd945a79ca5b739610704cbfa7019. 2023-04-08 02:56:44,067 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 087fd945a79ca5b739610704cbfa7019: 2023-04-08 02:56:44,067 DEBUG [RS_CLOSE_REGION-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:namespace,,1680922421094.087fd945a79ca5b739610704cbfa7019. 2023-04-08 02:56:44,240 DEBUG [RS:0;jenkins-hbase4:35089] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-08 02:56:44,441 DEBUG [RS:0;jenkins-hbase4:35089] regionserver.HRegionServer(1505): Waiting on 1588230740 2023-04-08 02:56:44,467 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=2.08 KB at sequenceid=93 (bloomFilter=false), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/1588230740/.tmp/table/b9adb3968f9e49e8bf7a4a4546d5adfe 2023-04-08 02:56:44,470 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b9adb3968f9e49e8bf7a4a4546d5adfe 2023-04-08 02:56:44,471 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/1588230740/.tmp/info/90b32090e52d4fd29067ee8baa366149 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/1588230740/info/90b32090e52d4fd29067ee8baa366149 2023-04-08 02:56:44,473 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 90b32090e52d4fd29067ee8baa366149 2023-04-08 02:56:44,473 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/1588230740/info/90b32090e52d4fd29067ee8baa366149, entries=22, sequenceid=93, filesize=7.1 K 2023-04-08 02:56:44,474 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/1588230740/.tmp/rep_barrier/78157665902b43cbb0f9f5a1708572ea as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/1588230740/rep_barrier/78157665902b43cbb0f9f5a1708572ea 2023-04-08 02:56:44,477 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for 78157665902b43cbb0f9f5a1708572ea 2023-04-08 02:56:44,477 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/1588230740/rep_barrier/78157665902b43cbb0f9f5a1708572ea, entries=6, sequenceid=93, filesize=5.5 K 2023-04-08 02:56:44,477 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/1588230740/.tmp/table/b9adb3968f9e49e8bf7a4a4546d5adfe as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/1588230740/table/b9adb3968f9e49e8bf7a4a4546d5adfe 2023-04-08 02:56:44,480 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for b9adb3968f9e49e8bf7a4a4546d5adfe 2023-04-08 02:56:44,480 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/1588230740/table/b9adb3968f9e49e8bf7a4a4546d5adfe, entries=9, sequenceid=93, filesize=5.5 K 2023-04-08 02:56:44,480 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(2948): Finished flush of dataSize ~20.92 KB/21417, heapSize ~36.20 KB/37064, currentSize=0 B/0 for 1588230740 in 842ms, sequenceid=93, compaction requested=false 2023-04-08 02:56:44,487 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] wal.WALSplitUtil(408): Wrote file=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/data/hbase/meta/1588230740/recovered.edits/96.seqid, newMaxSeqId=96, maxSeqId=1 2023-04-08 02:56:44,488 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] coprocessor.CoprocessorHost(310): Stop coprocessor org.apache.hadoop.hbase.coprocessor.MultiRowMutationEndpoint 2023-04-08 02:56:44,489 INFO [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1838): Closed hbase:meta,,1.1588230740 2023-04-08 02:56:44,489 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] regionserver.HRegion(1558): Region close journal for 1588230740: 2023-04-08 02:56:44,489 DEBUG [RS_CLOSE_META-regionserver/jenkins-hbase4:0-0] handler.CloseRegionHandler(117): Closed hbase:meta,,1.1588230740 2023-04-08 02:56:44,641 INFO [RS:0;jenkins-hbase4:35089] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,35089,1680922417983; all regions closed. 2023-04-08 02:56:44,649 DEBUG [RS:0;jenkins-hbase4:35089] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/oldWALs 2023-04-08 02:56:44,649 INFO [RS:0;jenkins-hbase4:35089] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C35089%2C1680922417983.meta:.meta(num 1680922420823) 2023-04-08 02:56:44,653 DEBUG [RS:0;jenkins-hbase4:35089] wal.AbstractFSWAL(1028): Moved 1 WAL file(s) to /user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/oldWALs 2023-04-08 02:56:44,653 INFO [RS:0;jenkins-hbase4:35089] wal.AbstractFSWAL(1031): Closed WAL: AsyncFSWAL jenkins-hbase4.apache.org%2C35089%2C1680922417983:(num 1680922419833) 2023-04-08 02:56:44,653 DEBUG [RS:0;jenkins-hbase4:35089] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:56:44,653 INFO [RS:0;jenkins-hbase4:35089] regionserver.LeaseManager(133): Closed leases 2023-04-08 02:56:44,653 INFO [RS:0;jenkins-hbase4:35089] hbase.ChoreService(369): Chore service for: regionserver/jenkins-hbase4:0 had [ScheduledChore name=CompactionThroughputTuner, period=60000, unit=MILLISECONDS, ScheduledChore name=CompactedHFilesCleaner, period=120000, unit=MILLISECONDS] on shutdown 2023-04-08 02:56:44,653 INFO [regionserver/jenkins-hbase4:0.logRoller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-08 02:56:44,655 INFO [RS:0;jenkins-hbase4:35089] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:35089 2023-04-08 02:56:44,660 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/rs/jenkins-hbase4.apache.org,35089,1680922417983 2023-04-08 02:56:44,660 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-08 02:56:44,660 ERROR [Listener at localhost/39741-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3905ae8d rejected from java.util.concurrent.ThreadPoolExecutor@3d3a2d75[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2503] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-08 02:56:44,661 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase/rs 2023-04-08 02:56:44,661 ERROR [Listener at localhost/39741-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@5d6f37da rejected from java.util.concurrent.ThreadPoolExecutor@3d3a2d75[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2503] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-08 02:56:44,662 INFO [RegionServerTracker-0] master.RegionServerTracker(179): RegionServer ephemeral node deleted, processing expiration [jenkins-hbase4.apache.org,35089,1680922417983] 2023-04-08 02:56:44,662 DEBUG [RegionServerTracker-0] master.DeadServer(103): Processing jenkins-hbase4.apache.org,35089,1680922417983; numProcessing=1 2023-04-08 02:56:44,664 DEBUG [RegionServerTracker-0] zookeeper.RecoverableZooKeeper(172): Node /hbase/draining/jenkins-hbase4.apache.org,35089,1680922417983 already deleted, retry=false 2023-04-08 02:56:44,664 INFO [RegionServerTracker-0] master.ServerManager(561): Cluster shutdown set; jenkins-hbase4.apache.org,35089,1680922417983 expired; onlineServers=0 2023-04-08 02:56:44,664 INFO [RegionServerTracker-0] regionserver.HRegionServer(2296): ***** STOPPING region server 'jenkins-hbase4.apache.org,46135,1680922416713' ***** 2023-04-08 02:56:44,664 INFO [RegionServerTracker-0] regionserver.HRegionServer(2310): STOPPED: Cluster shutdown set; onlineServer=0 2023-04-08 02:56:44,664 DEBUG [M:0;jenkins-hbase4:46135] ipc.AbstractRpcClient(190): Codec=org.apache.hadoop.hbase.codec.KeyValueCodec@eafb1fa, compressor=null, tcpKeepAlive=true, tcpNoDelay=true, connectTO=10000, readTO=20000, writeTO=60000, minIdleTimeBeforeClose=120000, maxRetries=0, fallbackAllowed=true, bind address=jenkins-hbase4.apache.org/172.31.14.131:0 2023-04-08 02:56:44,664 INFO [M:0;jenkins-hbase4:46135] regionserver.HRegionServer(1145): stopping server jenkins-hbase4.apache.org,46135,1680922416713 2023-04-08 02:56:44,664 INFO [M:0;jenkins-hbase4:46135] regionserver.HRegionServer(1171): stopping server jenkins-hbase4.apache.org,46135,1680922416713; all regions closed. 2023-04-08 02:56:44,664 DEBUG [M:0;jenkins-hbase4:46135] ipc.AbstractRpcClient(494): Stopping rpc client 2023-04-08 02:56:44,665 DEBUG [M:0;jenkins-hbase4:46135] cleaner.LogCleaner(198): Cancelling LogCleaner 2023-04-08 02:56:44,665 WARN [OldWALsCleaner-0] cleaner.LogCleaner(186): Interrupted while cleaning old WALs, will try to clean it next round. Exiting. 2023-04-08 02:56:44,665 DEBUG [M:0;jenkins-hbase4:46135] cleaner.HFileCleaner(317): Stopping file delete threads 2023-04-08 02:56:44,665 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680922419540] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.large.0-1680922419540,5,FailOnTimeoutGroup] 2023-04-08 02:56:44,665 DEBUG [master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680922419548] cleaner.HFileCleaner(288): Exit Thread[master/jenkins-hbase4:0:becomeActiveMaster-HFileCleaner.small.0-1680922419548,5,FailOnTimeoutGroup] 2023-04-08 02:56:44,665 INFO [M:0;jenkins-hbase4:46135] master.MasterMobCompactionThread(168): Waiting for Mob Compaction Thread to finish... 2023-04-08 02:56:44,665 INFO [M:0;jenkins-hbase4:46135] master.MasterMobCompactionThread(168): Waiting for Region Server Mob Compaction Thread to finish... 2023-04-08 02:56:44,665 INFO [M:0;jenkins-hbase4:46135] hbase.ChoreService(369): Chore service for: master/jenkins-hbase4:0 had [] on shutdown 2023-04-08 02:56:44,665 DEBUG [M:0;jenkins-hbase4:46135] master.HMaster(1512): Stopping service threads 2023-04-08 02:56:44,666 INFO [M:0;jenkins-hbase4:46135] procedure2.RemoteProcedureDispatcher(118): Stopping procedure remote dispatcher 2023-04-08 02:56:44,666 ERROR [M:0;jenkins-hbase4:46135] procedure2.ProcedureExecutor(653): ThreadGroup java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] contains running threads; null: See STDOUT java.lang.ThreadGroup[name=PEWorkerGroup,maxpri=10] Thread[HFileArchiver-13,5,PEWorkerGroup] 2023-04-08 02:56:44,666 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeDeleted, state=SyncConnected, path=/hbase/master 2023-04-08 02:56:44,666 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=NodeChildrenChanged, state=SyncConnected, path=/hbase 2023-04-08 02:56:44,666 INFO [M:0;jenkins-hbase4:46135] region.RegionProcedureStore(113): Stopping the Region Procedure Store, isAbort=false 2023-04-08 02:56:44,666 DEBUG [normalizer-worker-0] normalizer.RegionNormalizerWorker(174): interrupt detected. terminating. 2023-04-08 02:56:44,666 DEBUG [zk-event-processor-pool-0] zookeeper.ZKUtil(164): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Set watcher on znode that does not yet exist, /hbase/master 2023-04-08 02:56:44,667 DEBUG [M:0;jenkins-hbase4:46135] zookeeper.ZKUtil(398): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Unable to get data of znode /hbase/master because node does not exist (not an error) 2023-04-08 02:56:44,667 WARN [M:0;jenkins-hbase4:46135] master.ActiveMasterManager(326): Failed get of master address: java.io.IOException: Can't get master address from ZooKeeper; znode data == null 2023-04-08 02:56:44,667 INFO [M:0;jenkins-hbase4:46135] assignment.AssignmentManager(315): Stopping assignment manager 2023-04-08 02:56:44,667 INFO [M:0;jenkins-hbase4:46135] region.MasterRegion(167): Closing local region {ENCODED => 1595e783b53d99cd5eef43b6debb2682, NAME => 'master:store,,1.1595e783b53d99cd5eef43b6debb2682.', STARTKEY => '', ENDKEY => ''}, isAbort=false 2023-04-08 02:56:44,667 DEBUG [M:0;jenkins-hbase4:46135] regionserver.HRegion(1604): Closing 1595e783b53d99cd5eef43b6debb2682, disabling compactions & flushes 2023-04-08 02:56:44,667 INFO [M:0;jenkins-hbase4:46135] regionserver.HRegion(1626): Closing region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-08 02:56:44,667 DEBUG [M:0;jenkins-hbase4:46135] regionserver.HRegion(1647): Waiting without time limit for close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-08 02:56:44,667 DEBUG [M:0;jenkins-hbase4:46135] regionserver.HRegion(1714): Acquired close lock on master:store,,1.1595e783b53d99cd5eef43b6debb2682. after waiting 0 ms 2023-04-08 02:56:44,667 DEBUG [M:0;jenkins-hbase4:46135] regionserver.HRegion(1724): Updates disabled for region master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-08 02:56:44,668 INFO [M:0;jenkins-hbase4:46135] regionserver.HRegion(2745): Flushing 1595e783b53d99cd5eef43b6debb2682 1/1 column families, dataSize=344.86 KB heapSize=393.59 KB 2023-04-08 02:56:44,763 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-08 02:56:44,763 INFO [RS:0;jenkins-hbase4:35089] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,35089,1680922417983; zookeeper connection closed. 2023-04-08 02:56:44,763 ERROR [Listener at localhost/39741-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@25b756ed rejected from java.util.concurrent.ThreadPoolExecutor@3d3a2d75[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2503] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-08 02:56:44,764 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): regionserver:35089-0x10161b07ca90001, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-08 02:56:44,764 INFO [Shutdown of org.apache.hadoop.hbase.fs.HFileSystem@66c6bdec] hbase.MiniHBaseCluster$SingleFileSystemShutdownThread(215): Hook closing fs=org.apache.hadoop.hbase.fs.HFileSystem@66c6bdec 2023-04-08 02:56:44,764 ERROR [Listener at localhost/39741-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@3c627ec6 rejected from java.util.concurrent.ThreadPoolExecutor@3d3a2d75[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 2503] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-08 02:56:44,764 INFO [Listener at localhost/39741] util.JVMClusterUtil(335): Shutdown of 1 master(s) and 1 regionserver(s) complete 2023-04-08 02:56:45,086 INFO [M:0;jenkins-hbase4:46135] regionserver.DefaultStoreFlusher(82): Flushed memstore data size=344.86 KB at sequenceid=578 (bloomFilter=true), to=hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/c39834b3cf734417986cbb9141437297 2023-04-08 02:56:45,089 INFO [M:0;jenkins-hbase4:46135] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c39834b3cf734417986cbb9141437297 2023-04-08 02:56:45,090 DEBUG [M:0;jenkins-hbase4:46135] regionserver.HRegionFileSystem(485): Committing hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/.tmp/proc/c39834b3cf734417986cbb9141437297 as hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/c39834b3cf734417986cbb9141437297 2023-04-08 02:56:45,092 INFO [M:0;jenkins-hbase4:46135] regionserver.StoreFileReader(520): Loaded Delete Family Bloom (CompoundBloomFilter) metadata for c39834b3cf734417986cbb9141437297 2023-04-08 02:56:45,092 INFO [M:0;jenkins-hbase4:46135] regionserver.HStore(1080): Added hdfs://localhost:42901/user/jenkins/test-data/240f94fe-857b-62d3-530a-8b3d1484ef01/MasterData/data/master/store/1595e783b53d99cd5eef43b6debb2682/proc/c39834b3cf734417986cbb9141437297, entries=65, sequenceid=578, filesize=24.7 K 2023-04-08 02:56:45,093 INFO [M:0;jenkins-hbase4:46135] regionserver.HRegion(2948): Finished flush of dataSize ~344.86 KB/353133, heapSize ~393.57 KB/403016, currentSize=0 B/0 for 1595e783b53d99cd5eef43b6debb2682 in 426ms, sequenceid=578, compaction requested=false 2023-04-08 02:56:45,095 INFO [M:0;jenkins-hbase4:46135] regionserver.HRegion(1838): Closed master:store,,1.1595e783b53d99cd5eef43b6debb2682. 2023-04-08 02:56:45,095 DEBUG [M:0;jenkins-hbase4:46135] regionserver.HRegion(1558): Region close journal for 1595e783b53d99cd5eef43b6debb2682: 2023-04-08 02:56:45,097 INFO [master:store-WAL-Roller] wal.AbstractWALRoller(243): LogRoller exiting. 2023-04-08 02:56:45,097 INFO [M:0;jenkins-hbase4:46135] flush.MasterFlushTableProcedureManager(83): stop: server shutting down. 2023-04-08 02:56:45,098 INFO [M:0;jenkins-hbase4:46135] ipc.NettyRpcServer(158): Stopping server on /172.31.14.131:46135 2023-04-08 02:56:45,099 DEBUG [M:0;jenkins-hbase4:46135] zookeeper.RecoverableZooKeeper(172): Node /hbase/rs/jenkins-hbase4.apache.org,46135,1680922416713 already deleted, retry=false 2023-04-08 02:56:45,201 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-08 02:56:45,201 INFO [M:0;jenkins-hbase4:46135] regionserver.HRegionServer(1228): Exiting; stopping=jenkins-hbase4.apache.org,46135,1680922416713; zookeeper connection closed. 2023-04-08 02:56:45,201 ERROR [Listener at localhost/39741-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@64880970 rejected from java.util.concurrent.ThreadPoolExecutor@5da185c7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4184] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-08 02:56:45,202 DEBUG [Listener at localhost/39741-EventThread] zookeeper.ZKWatcher(600): master:46135-0x10161b07ca90000, quorum=127.0.0.1:64826, baseZNode=/hbase Received ZooKeeper Event, type=None, state=Closed, path=null 2023-04-08 02:56:45,202 ERROR [Listener at localhost/39741-EventThread] zookeeper.ClientCnxn$EventThread(537): Error while calling watcher java.util.concurrent.RejectedExecutionException: Task java.util.concurrent.FutureTask@339cf1e1 rejected from java.util.concurrent.ThreadPoolExecutor@5da185c7[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 4184] at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063) at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830) at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379) at java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:112) at java.util.concurrent.Executors$DelegatedExecutorService.submit(Executors.java:678) at org.apache.hadoop.hbase.zookeeper.ZKWatcher.process(ZKWatcher.java:602) at org.apache.hadoop.hbase.zookeeper.PendingWatcher.process(PendingWatcher.java:38) at org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:535) at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:510) 2023-04-08 02:56:45,203 WARN [Listener at localhost/39741] datanode.DirectoryScanner(534): DirectoryScanner: shutdown has been called 2023-04-08 02:56:45,206 INFO [Listener at localhost/39741] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-08 02:56:45,314 WARN [BP-1518829044-172.31.14.131-1680922413296 heartbeating to localhost/127.0.0.1:42901] datanode.IncrementalBlockReportManager(160): IncrementalBlockReportManager interrupted 2023-04-08 02:56:45,314 WARN [BP-1518829044-172.31.14.131-1680922413296 heartbeating to localhost/127.0.0.1:42901] datanode.BPServiceActor(857): Ending block pool service for: Block pool BP-1518829044-172.31.14.131-1680922413296 (Datanode Uuid 14d5a713-17f1-418e-9bd0-2301d2c6ae84) service to localhost/127.0.0.1:42901 2023-04-08 02:56:45,317 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b/cluster_2b175f16-edf4-6ecc-e1c4-da3bf5a7233b/dfs/data/data1/current/BP-1518829044-172.31.14.131-1680922413296] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-08 02:56:45,318 WARN [refreshUsed-/home/jenkins/jenkins-home/workspace/HBase-Flaky-Tests_branch-2.4/hbase-server/target/test-data/aca5d7cb-8588-2019-00c2-3722724e214b/cluster_2b175f16-edf4-6ecc-e1c4-da3bf5a7233b/dfs/data/data2/current/BP-1518829044-172.31.14.131-1680922413296] fs.CachingGetSpaceUsed$RefreshThread(183): Thread Interrupted waiting to refresh disk information: sleep interrupted 2023-04-08 02:56:45,350 INFO [Listener at localhost/39741] log.Slf4jLog(67): Stopped HttpServer2$SelectChannelConnectorWithSafeStartup@localhost:0 2023-04-08 02:56:45,541 INFO [Listener at localhost/39741] zookeeper.MiniZooKeeperCluster(344): Shutdown MiniZK cluster with all ZK servers 2023-04-08 02:56:45,730 INFO [Listener at localhost/39741] hbase.HBaseTestingUtility(1293): Minicluster is down